Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re: Perl Code Runs Extremely Slow

by samtregar (Abbot)
on Jun 13, 2006 at 20:06 UTC ( #555116=note: print w/ replies, xml ) Need Help??


in reply to Perl Code Runs Extremely Slow

There are so many performance problems in this code that it's kind of hard to know where to begin! Here's a few that jump out right away:

  • Don't open file 2 for each line of file 1 and read through every line! If there are 1 million lines in file 1 and 500 thousand lines in file 2 then you'll read 500 billion lines from file 2! Instead read file 2 once and re-use the hash for each lookup.
  • Don't re-sort all the keys from file 1 everytime you read a line from file 1. (UPDATE: Looking again I see that %fets is actually local to the while(). Why are you using a hash at all here? Why are you calling sort() when only one key is present?)
  • You may not have enough memory to actually hold all of file 2 in memory at once. If you don't you'll run into swap, which will be slow no matter what you do. You can fix this by storing the hash in a database file via DB_File or something similar.

-sam


Comment on Re: Perl Code Runs Extremely Slow

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://555116]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (8)
As of 2015-07-05 18:37 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (67 votes), past polls