Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: Using less memory with BIG files

by moritz (Cardinal)
on Feb 02, 2012 at 08:11 UTC ( #951373=note: print w/ replies, xml ) Need Help??


in reply to Using less memory with BIG files

There are a number of things you can improve. The two most important are:

1. If you do foreach $unit(<PFAMDB>) { } it reads the whole file into memory first, then iterates over it. If you instead write while ($unit = <PFAMDB>) { ... }, the file is read line by line.

2.Instead of doing a nested loop, read the IDs into a hash first, and then extract the IDs from the second file, and look them up in the hash. That will greatly speed up things.


Comment on Re: Using less memory with BIG files
Select or Download Code
Re^2: Using less memory with BIG files
by jemswira (Novice) on Feb 02, 2012 at 12:11 UTC

    for the second part, I store each line as a hash key, then I don't really know what to do after that?

    would it make more sense if i set each group to a hash with the key being the group name and the value being the number? problem is there would be repeats for the IDs so i don't know what can i do.

    sorry but im still kinda new to this D:

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://951373]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (6)
As of 2014-07-31 00:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (241 votes), past polls