Multiple substitutions in large files

[mdi]

I need to do multiple substitutions in several large (1-10MB) files. I've been using this:

use strict;
use warnings;
use Tie::File;

foreach my $x (@ARGV) {
    tie my @f, 'Tie::File', $x or die "Could not tie $x: $!\n";
    for (@f) {
        s/^\|/\\N\|/;
        s/\|\s*$/\|\\N/;
        s/\|\s*\|/\|\\N\|/g;
        s/\|\.\s*\|/\|\\N\|/g;
        s/\|\s+/\|/g;
        s/\s+\|/\|/g;
        s/(\d{2}:\d{2}:\d{2})\.\d+/$1/g;
        s/(\d{5})-(?:\d{1,4}|\s+)/$1/;
    }
}

...but this is taking entirely too long, and using up too much CPU. How can I do this more efficiently?

[dragonchild]

#!/usr/bin/perl -p

        s/^\|/\\N\|/;
        s/\|\s*$/\|\\N/;
        s/\|\s*\|/\|\\N\|/g;
        s/\|\.\s*\|/\|\\N\|/g;
        s/\|\s+/\|/g;
        s/\s+\|/\|/g;
        s/(\d{2}:\d{2}:\d{2})\.\d+/$1/g;
        s/(\d{5})-(?:\d{1,4}|\s+)/$1/;

Execute as so:

my_scriptydoo.pl file1 > file2

Update: ikegami is absolutely correct. I should be doing a redirect. The next 1st level response provides the -pi version.

[ikegami]

Shouldn't that be -pi (or -pi.bak if a backup is desired)? With just -p, the usage would be my_scriptydoo.pl file1 > file1.new

[Joost]

#!/usr/bin/perl -pi

        s/^\|/\\N\|/;
        s/\|\s*$/\|\\N/;
        s/\|\s*\|/\|\\N\|/g;
        s/\|\.\s*\|/\|\\N\|/g;
        s/\|\s+/\|/g;
        s/\s+\|/\|/g;
        s/(\d{2}:\d{2}:\d{2})\.\d+/$1/g;
        s/(\d{5})-(?:\d{1,4}|\s+)/$1/;

Should be very much more effective, because it creates new files instead of trying to move all the bytes in the file around for each substitution.

See perlrun's info on the -i and -p switches.

[ikegami]

a|b||d becomes a|b|\N|d
|b|c|d becomes \N|b|c|d
a|b|c| becomes a|b|c|\N
and similarly,
a|b|.|d becomes a|b|\N|d
but
.|b|c|d does not become \N|b|c|d
a|b|c|. does not become a|b|c|\N
Is that a bug?

If the above is a bug, the following regexps are probably faster:

s/\s*\|\s*/\|/g;
s/^\.?(?=\|)/\\N/;
s/(?<=\|)\.?(?=\||$)/\\N/g;
s/(?<=\d{2}:\d{2}:\d{2})\.\d+//g;
s/(?<=\d{5})-(?:\d{1,4}|\s+)//;

If the above is not a bug, the following regexps are probably faster:

s/\s*\|\s*/\|/g;
s/^(?=\|)/\\N/;
s/(?<=\|)(?=\||$)/\\N/g;
s/(?<=\|)\.(?=\|)/\\N/g;
s/(?<=\d{2}:\d{2}:\d{2})\.\d+//g;
s/(?<=\d{5})-(?:\d{1,4}|\s+)//;

I reduced the number of regexps by combining a few, I shortened the regexps by removing the spaces first (not last), and I used zero-widths positive lookaheads and lookbehinds to mimimze the text being captured and substituted.

Use this in conjuction with the -p or -pi suggestion for better results.