Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?

Re: Re: combine multiple files into one (line by line)

by Anonymous Monk
on Apr 05, 2001 at 22:53 UTC ( #70225=note: print w/ replies, xml ) Need Help??

in reply to Re: combine multiple files into one (line by line)
in thread combine multiple files into one (line by line)

Thank you for your help, however I failed to mention that the files range in size from 55MB to 130MB. In addition, I only want to grab the first 2,000 or so lines from each file and distribute them in the abovementioned fashion.

For these reasons I don't think that using arrays is feasible. I understand that opening and closing is expensive, but hopefully less so than using huge arrays.

Thanks for your help,


p.s. thabenksta's code works, but yours does not.

Comment on Re: Re: combine multiple files into one (line by line)
Replies are listed 'Best First'.
Re: Re: Re: combine multiple files into one (line by line)
by extremely (Priest) on Apr 05, 2001 at 23:36 UTC
    I don't think he was intending to load the entire file into memory. The trick is to keep an array of open FILEHANDLES and read one line at a time from each...
    #!/usr/bin/perl -w use strict; use IO::File; #interleave files, first argument is line count, rest are files. my $lc = shift; my @fhs; foreach (@ARGV) { my $fh = new IO::File; open ($fh, "<$_"); push @fhs, $fh; } while (--$lc) { foreach (@fhs) { print scalar(<$_>); } }

    Tested minimally but the path is clear from here I hope.

    $you = new YOU;
    honk() if $you->love(perl)

Re: Re: Re: combine multiple files into one (line by line)
by jeroenes (Priest) on Apr 05, 2001 at 23:49 UTC
    You really don't need to read all files in. For clarity I will use filehandles.
    use FileHandle; my @files = qw( 1.txt 2.txt 3.txt); my @handles = map{ $_ = new FileHandle $_, 'r'; die "Could not open $_: $!" unless defined($_); $_; } @files; open MIXED, ">$mixed_file" or die "Could not open $mixed_file: $!"; while( @handles ){ foreach my $fh (@handles){ my $line = $fh->getline or undef $fh; print MIXED $line if defined $fh; } @handles = grep defined, @handles; } close MIXED;
    Hope this helps,

    "We are not alone"(FZ)

Re: Re: Re: combine multiple files into one (line by line)
by ton (Friar) on Apr 06, 2001 at 00:01 UTC
    Whoops, that's what happens when you submit code without testing it first :). There were four things wrong with my code:
    1. The open call for the MIXED file did not have a '>'. This is sheer carelessness.
    2. I was openning the files to the same typeglob, which resulted in three references to the same open file being stored in the array. This is a more subtle bug.
    3. I was testing for eof after reading in a line, instead of before. This would result in the last line never being written to the mixed file.
    4. Finally, I was using 'last' within the inner loop, foolishly thinking it would get me out of the outer loop. Another dumb mistake.
    Working code (at least on my Wintel box) follows:
    my @files = ("1.txt", "2.txt", "3.txt"); my @fhs; for ($i = 0; $i < scalar(@files); ++$i) { open($fhs[$i], $files[$i]) || die; } open(MIXED, ">mixed.txt") || die; OUTER: while (1) { foreach $fh (@fhs) { last OUTER if (eof($fh)); $line = <$fh>; print MIXED $line; } } map { close($_); } @fhs; close(MIXED);
    Let this be a lesson for me in preliminary testing!


Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://70225]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (4)
As of 2015-10-07 22:27 GMT
Find Nodes?
    Voting Booth?

    Does Humor Belong in Programming?

    Results (202 votes), past polls