Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl: the Markov chain saw
 
PerlMonks  

Re^3: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)

by Tanktalus (Canon)
on Nov 14, 2012 at 04:35 UTC ( #1003748=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)
in thread Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)

I don't see how that would work. You're proposing fixing a performance issue caused by fork/exec/re-reading a large file from the beginning on each iteration, by doing exactly the same thing but with a virtual machine added in the middle instead of optimised C code? This isn't a perl-specific question you're asking, it's fairly generic. The proposed solutions (e.g., Tie::File) include some perl-specific suggestions, and some that aren't (read the file into memory as a list/array - you can do that in C++ with the STL fairly easily, and Java should make it pretty simple, too), but the general issue is language-agnostic.

Instead, if you read it all into memory, you can use likely just a line to duplicate. Without testing or even compiling:

# do this once. OUTSIDE OF YOUR LOOP. my @read_source_lines = do { open my $fh, '<', $read_source or die "Can't read from $read_source: + $!"; <$fh> }; # you may also need: chomp @read_source_lines; # gets rid of \n's. # inside the loop, instead of $strx/$str5: my $str5 = $read_source_lines[$recnum]; print REPORT "$_|$recnum$str5\n";
Assuming you don't start swapping, this should eliminate most of your time. Note that there are better/faster ways to do this, but this will get you most of the benefit for the least amount of effort. Many of those better ways are actually embedded in Tie::File, IIRC (reading only as many lines as is currently needed, continuing from where you left off, maybe you don't need to read the entire file, this may also allow the OS to continue reading the file in the background to fill up your input buffers while you go do other work, that type of thing).


Comment on Re^3: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)
Download Code
Re^4: Need help to fine tune perl script to make it faster( currently taking more than 30 minutes)
by anujajoseph (Novice) on Nov 14, 2012 at 09:48 UTC
    Thank you so much for your genuine help and time! Appreciate much! This piece of code you have suggested( without using Tie::File) , works perfectly fine. now the whole processing takes only a second. I have heard always perl is very fast, now saw its performance. thanks again for making me a PERL fan too :-)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1003748]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others exploiting the Monastery: (8)
As of 2014-12-23 05:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (135 votes), past polls