Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: About text file parsing

by SuicideJunkie (Vicar)
on Aug 29, 2018 at 17:42 UTC ( #1221316=note: print w/replies, xml ) Need Help??


in reply to About text file parsing

XY problem question here...
What are you going to be using those arrays for? Are they huge, or is the sample/good a small subset of the input data? There may be better ways to approach the entire task.

Perhaps consider something more like this:

use strict; use warnings; open my $ifh, '<', 'test.pl' or die; open my $ofh_samples, '>', 'samples.txt' or die; open my $ofh_good, '>', 'goodlines.txt' or die; $|=1; my $total = -s 'test.pl'; my $progress = 0; my $linecount = 0; while (my $line = <$ifh>) { $linecount++; $progress += length ($line); print $ofh_samples $1 if $line =~ /^sample\s+(\S+)/; print $ofh_good $1 if $line =~ /^good\s+(\S+)/; printf "Processing... %3.1f%% completed... \r", 100*$progress/$tot +al unless $linecount %100; }
That will keep little more than one line in memory at a time, and you can then deal with the pieces separately. Uncomment the comments to give a progress display.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1221316]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (5)
As of 2020-11-23 17:40 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found

    Notices?