Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re: A very odd happening (at least. . . to me)

by maverick (Curate)
on Jun 24, 2002 at 16:09 UTC ( #176843=note: print w/replies, xml ) Need Help??

in reply to A very odd happening (at least. . . to me)
in thread Processing large files many times over

From a quick glance at the code, one of the first questions that comes to mind is "how many files are in these directories?" I suspect that part of the source of your slowness is that you read both the entire list of files, and the entire contents of each file into memory. If you alter your reading structure like so:
open(DIR,"$base_dir\\$dir") or die "$dir failed to open: $!"; while (my $file = readdir(DIR)) { next unless $file =~ /\.txt$/; # etc, etc. open(IN,"$full_name") || die "can't open $!"; while (my $line = <IN>) { # processing } close(IN); } closedir(DIR);
you won't have the overhead of all the memory allocation. In your second example there's a system call to a secondary perl script. That's going to be time consuming too. Consider making the second perl program a subroutine...that will avoid a fork, exec, and compile for every file you have.


OmG! They killed tilly! You *bleep*!!

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://176843]
[Corion]: LanX: Ah, yeah - Frankfurt is in the quake region, but at the very border. I think I've never noticed a quake in Frankfurt myself
[marto]: good morning all
[Corion]: On another topic, in the process of Rubber Duck SoPWing, I wrote a post about the best API for generating HTTP requests (not sending them) and while writing it and fleshing out my sample implementation, I came up with some improvements to my ...
[Corion]: ... existing prototype. Cartesian Products will be proud of that module once it gets out ;)
[Corion]: Hi marto ;)

How do I use this? | Other CB clients
Other Users?
Others taking refuge in the Monastery: (5)
As of 2017-01-18 10:13 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (161 votes). Check out past polls.