Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Using less memory with BIG files

by moritz (Cardinal)
on Feb 02, 2012 at 08:11 UTC ( #951373=note: print w/ replies, xml ) Need Help??


in reply to Using less memory with BIG files

There are a number of things you can improve. The two most important are:

1. If you do foreach $unit(<PFAMDB>) { } it reads the whole file into memory first, then iterates over it. If you instead write while ($unit = <PFAMDB>) { ... }, the file is read line by line.

2.Instead of doing a nested loop, read the IDs into a hash first, and then extract the IDs from the second file, and look them up in the hash. That will greatly speed up things.


Comment on Re: Using less memory with BIG files
Select or Download Code
Re^2: Using less memory with BIG files
by jemswira (Novice) on Feb 02, 2012 at 12:11 UTC

    for the second part, I store each line as a hash key, then I don't really know what to do after that?

    would it make more sense if i set each group to a hash with the key being the group name and the value being the number? problem is there would be repeats for the IDs so i don't know what can i do.

    sorry but im still kinda new to this D:

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://951373]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (12)
As of 2015-07-06 15:54 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (77 votes), past polls