Re^3: Help me beat NodeJS

[rickyw59]

Wow thanks, I'll give this a shot. I will have to read some more on MCE, it looks very useful. I should have clarified, in the "parser" function in nodejs, I'm applying the same regex as perl to be fair. I've done the tests looking for a specific string (/some_string/) and I've done the regex in the above code (/".*?"|\S+/g), which captures everything in an array, since the lines are in this format: ' 1970-01-01 00:00:00 1.1.1.1 "A multi-word field" 2.2.2.2 '

[marioroy]

Got it. I went ahead and updated both MCE demonstrations to account for pattern matching. The more expensive regex (/".*?"|\S+/g) pattern is processed only if given line matches the initial string pattern. That will likely run faster.

Likewise, for Parallel::ForkManager.

#!/usr/local/bin/perl

use strict;
use warnings;
use Parallel::ForkManager;

my $pm = new Parallel::ForkManager(24);

my $dir = '/data/logs/*.log.gz';
my @files = sort(glob "$dir");
my $pattern = "some_string";

$pm->set_waitpid_blocking_sleep(0);

for my $file( @files ) {
    $pm->start and next;
    open( my $fh, "-|", "/bin/zcat", $file ) or die "open error: $!\n"
+;
    while ( my $line = <$fh> ) {
        if ( $line =~ /$pattern/ ) {
            my @matches = $line =~ /".*?"|\S+/g;
            print "$matches[0],$matches[1],$matches[3],$matches[4]\n";
        }
    }
    $pm->finish;
}

$pm->wait_all_children;

Regards, Mario

[RichardK]

If you just looking for a plain string not a regex, then it should be quicker to use index

while (my $line = <$fh> ) {
    if (index($line,$pattern) != -1 ) {
        ...
    }
}

[marioroy]

yes, ++