large file issue?

[clearcache]

this is the first time I'm posting a question, so I apologize if my etiquette isn't quite right...

I am performing a simple search-and-replace on a large file - a very large file - already in an array and writing out to a new file. Then, I have a few system() statements that manipulate files. The problem is, my program seems to be skipping my system() statements. My instinct tells me it is because of the file size, but I don't know how to fix it. My code looks like something like this:

open(OUT,">outfile") || die;
for($i=0;$i<=$#bigfile;$i++){
  $bigfile[$i] =~ s/\0/ /g;
  print OUT "$bigfile[$i]";
}
close(OUT);
print "Compressing original file\n";
system("compress original.file");

print "Concatenating other data\n";
system("cat ./incoming/data.file ./static/data.file >> outfile");

print "Sorting Data File\n";
system("asort outfile clean.data 1 6 8 9");

print "Done.\n";

The odd thing is that it performs the print statements correctly, but the system statements are all skipped. Any ideas? (part of the reason my instinct tells me that it is filesize is that it has worked fine for the entire year up until now - and that's the only variable I can think of that may affect this)

[Shendal]

Whenever you use system commands, you will want to make sure you check the exit status. Additionally, the error is stored in $!, and should give you an idea of why things are failing. Try something like this:


open(OUT,">outfile") || die;
for($i=0;$i<=$#bigfile;$i++){
  $bigfile[$i] =~ s/\0/ /g;
  print OUT "$bigfile[$i]";
}
close(OUT);
print "Compressing original file\n";
system("compress original.file")
  and die "Unable to compress data: $!\n";

print "Concatenating other data\n";
system("cat ./incoming/data.file ./static/data.file >> outfile")
  and die "Unable to cat data: $!\n";

print "Sorting Data File\n";
system("asort outfile clean.data 1 6 8 9")
  and die "Unable to sort data: $!\n";

print "Done.\n";

You may also want to use system in a list context, as it is faster and more secure, for example:


system("/path/to/asort","outfile","clean.data","1 6 8 9")
  and die "Unable to sort data: $!\n";

Lastly, your for loop isn't very perlish. You may want to try something like this:

open(OUT,">outfile") || die "Unable to open outfile: $!\n";
foreach (@bigfile) {
  s/\0/ /g;
  print OUT "$_";
}
close(OUT);


Cheers,
Shendal

Update: As runrig pointed out via the chatterbox, it ought to be and die after a system, since system returns 0 on success.

Update2: See merlyn's post RE: Re: large file issue?. $! is actually not what you want to look at.

[merlyn]

Nope. You don't want to be looking at $! after system. The child exit status is in $? (as well as being the return value from system), and the child's reason for exiting is not available to you in any way shape or form. We went through this with tilly a few weeks ago. {grin}

-- Randal L. Schwartz, Perl hacker

[clearcache]

thanks for the quick reply - I tried system(xxx) || die "$!" and it didn't tell me anything. ...and thanks for the system in list context hint - I had never thought of that. and the loop - yeah, I know...the loop is actually doing a lot more - I just took out some code to avoid identifying my employer/client ;)

[amelinda]

If you're going to try that, try system("blah") or die "$!\n"; instead... || has higher precedence and screws things up.

[clearcache]

Thanks to all who helped.

When I added the && die "$!", I was returned with the error "Not enough space". Some quick searching told me that I didn't have enough swap space on the machine and I changed my code to use a while loop for the null-character substitution - a later loop takes care of everything else. The file is just so damn big that the array forces the UNIX box to use swap space...swap space that I didn't have.

Thanks.

[jynx]

Just a thought,

Since you have a large file, rather than slurping the whole file to an array, is there a way you could just read through the file and still get everything you need to do completed? It would save you from needing a lot of memory and you wouldn't have the errors on the system calls (in theory).

jynx