Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change

Re: Processing while reading in input

by LanX (Archbishop)
on Sep 20, 2018 at 00:10 UTC ( #1222682=note: print w/replies, xml ) Need Help??

in reply to Processing while reading in input

Update: The following reply deals with unsorted input. I didn't expect such a trivial case ...

Your main problem is generating the output file, I'd suggest generating one temporary file per cluster and merging them at the end *. Like this you just need to append to the temporary files and keep track of the clusters.

Now processing the input, you could use a sliding window to process big chunks (like 100MB), but I don't think it'll make a big difference from reading line by line with readline (since main limitation is hard-disk speed here and Perl and OS are already reading in big chunks behind the scene).

But I would certainly group the write operations, like processing n = 1 million lines before writing out. Collect the entries in a hash of arrays push @{$hash{$cluster}}, $entry and append them to the temporary cluster files ( open has an append mode with '>>' ). Then empty the hash to avoid memory problems and process the next n lines.

NB: In case the entries have to be unique within a cluster (you haven't been precise about that) you'd need a hash of hashes and a more complicated approach.


Cheers Rolf
(addicted to the Perl Programming Language :)
Wikisyntax for the Monastery FootballPerl is like chess, only without the dice

*) I'm not sure about the most efficient way OS-wise to merge large files, but google or the monastery should know. I'm critical about this obsession of you bio-guys of creating huge files. I'd rather have data separated into several smaller files and zipped them together.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1222682]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others contemplating the Monastery: (3)
As of 2019-07-20 05:01 GMT
Find Nodes?
    Voting Booth?

    No recent polls found