Beefy Boxes and Bandwidth Generously Provided by pair Networks
go ahead... be a heretic

Merge huge files (individually sorted) by order

by tanger007 (Initiate)
on Jul 18, 2013 at 23:55 UTC ( #1045229=perlquestion: print w/replies, xml ) Need Help??
tanger007 has asked for the wisdom of the Perl Monks concerning the following question:

Been trying to merge a bunch of sorted files with the same ordering. The number of files is random, so I chose to use an array of file handles. The plan is to have each file send in one line, then, the line with the highest rank will get pushed out, and then another line is yanked out of the file where that last line was from... How does this work with perl? Been trying 3 hours with no success...
  • Comment on Merge huge files (individually sorted) by order

Replies are listed 'Best First'.
Re: Merge huge files (individually sorted) by order
by Loops (Curate) on Jul 18, 2013 at 23:57 UTC
      Works so well I felt more stupid :) A follow up question: if you have a big file (>10GB) in which one column has say 100 unique values. How do you break this file into 100 smaller files with one unique value in that column? Thanks so much.


        Try something like putting a file handle for each column value in a hash, and then looking up the file handle on demand:

        my %OFH; my $OFH; while (<$IFH>) { my @fields = split /\t/,$_; $OFH = $OFH{$fields[$key_column]}; if (! defined $OFH) { # We don't have this value yet, so open another file open $OFH, '>', 'key_value.' . $fields[$key_column]; $OFH{$fields[$key_column]} = $OFH; } print $OFH join("\t",@fields); }

        Note: It's rough, untested and needs some error handling and such. But the basic concept should work fine for you.


        When your only tool is a hammer, all problems look like your thumb.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1045229]
Approved by toolic
Front-paged by toolic
[stevieb]: LOL you can easily replace "Helsinki" with "Alberta"
[perldigious]: I used to live at the top of a very steep hill stevieb, I can relate to the video. Literally used to have to build up momentum and try to force my vehicle to make it up the hill to park in my driveway properly... didn't always make it.
[choroba]: Why don't they use tire chains?
[perldigious]: First snow, probably not prepared right away choroba.
[stevieb]: it's Quebec... they do weird stuff there :) In the mountains, it is *mandatory* for trucks to use tire chains. Many passenger vehicles do as well (but it's not mandatory for them)
[perldigious]: That stuff can hit fast... like what we just got where I'm at. Saw a few people who had slid OUT of the roundabout I have to drive through to get to work, not used to the ice yet, take awhile to get the hang of it again and make vehicle adjustments.
[stevieb]: man, when the roads are glare ice, I don't even bother going to work or out... unless I absolutely have to, or was already out in the first place
[choroba]: Tire chains are mandatory here in mountains, and the only unprepared each year are gritters
[stevieb]: I have a 3 day winter survival kit in my vehicle in the event I get snowed in in the mountains (which has happened before due to avalanches closing the roads (and once in the summer due to a massive forest fire that trapped us
[perldigious]: a handful of people in my work area did not make it, but I live pretty close and it's flat the whole way, so I didn't have any trouble. The roundabout is the worst thing I have to negotiate.

How do I use this? | Other CB clients
Other Users?
Others pondering the Monastery: (9)
As of 2016-12-06 16:36 GMT
Find Nodes?
    Voting Booth?
    On a regular basis, I'm most likely to spy upon:

    Results (112 votes). Check out past polls.