Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Re: Re: combine multiple files into one (line by line)

by Anonymous Monk
on Apr 05, 2001 at 22:53 UTC ( #70225=note: print w/ replies, xml ) Need Help??


in reply to Re: combine multiple files into one (line by line)
in thread combine multiple files into one (line by line)

Thank you for your help, however I failed to mention that the files range in size from 55MB to 130MB. In addition, I only want to grab the first 2,000 or so lines from each file and distribute them in the abovementioned fashion.

For these reasons I don't think that using arrays is feasible. I understand that opening and closing is expensive, but hopefully less so than using huge arrays.

Thanks for your help,

Roger

p.s. thabenksta's code works, but yours does not.


Comment on Re: Re: combine multiple files into one (line by line)
Re: Re: Re: combine multiple files into one (line by line)
by extremely (Priest) on Apr 05, 2001 at 23:36 UTC
    I don't think he was intending to load the entire file into memory. The trick is to keep an array of open FILEHANDLES and read one line at a time from each...
    #!/usr/bin/perl -w use strict; use IO::File; #interleave files, first argument is line count, rest are files. my $lc = shift; my @fhs; foreach (@ARGV) { my $fh = new IO::File; open ($fh, "<$_"); push @fhs, $fh; } while (--$lc) { foreach (@fhs) { print scalar(<$_>); } }

    Tested minimally but the path is clear from here I hope.

    --
    $you = new YOU;
    honk() if $you->love(perl)

Re: Re: Re: combine multiple files into one (line by line)
by jeroenes (Priest) on Apr 05, 2001 at 23:49 UTC
    You really don't need to read all files in. For clarity I will use filehandles.
    use FileHandle; my @files = qw( 1.txt 2.txt 3.txt); my @handles = map{ $_ = new FileHandle $_, 'r'; die "Could not open $_: $!" unless defined($_); $_; } @files; open MIXED, ">$mixed_file" or die "Could not open $mixed_file: $!"; while( @handles ){ foreach my $fh (@handles){ my $line = $fh->getline or undef $fh; print MIXED $line if defined $fh; } @handles = grep defined, @handles; } close MIXED;
    Hope this helps,

    Jeroen
    "We are not alone"(FZ)

Re: Re: Re: combine multiple files into one (line by line)
by ton (Friar) on Apr 06, 2001 at 00:01 UTC
    Whoops, that's what happens when you submit code without testing it first :). There were four things wrong with my code:
    1. The open call for the MIXED file did not have a '>'. This is sheer carelessness.
    2. I was openning the files to the same typeglob, which resulted in three references to the same open file being stored in the array. This is a more subtle bug.
    3. I was testing for eof after reading in a line, instead of before. This would result in the last line never being written to the mixed file.
    4. Finally, I was using 'last' within the inner loop, foolishly thinking it would get me out of the outer loop. Another dumb mistake.
    Working code (at least on my Wintel box) follows:
    my @files = ("1.txt", "2.txt", "3.txt"); my @fhs; for ($i = 0; $i < scalar(@files); ++$i) { open($fhs[$i], $files[$i]) || die; } open(MIXED, ">mixed.txt") || die; OUTER: while (1) { foreach $fh (@fhs) { last OUTER if (eof($fh)); $line = <$fh>; print MIXED $line; } } map { close($_); } @fhs; close(MIXED);
    Let this be a lesson for me in preliminary testing!

    -ton

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://70225]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2014-09-18 23:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (126 votes), past polls