Re^3: Searching large files a block at a time

[kcott]

"... helped me understand what I was doing wrong ..."

OK, that's a good start.

"Using your code, I get results in ~10 seconds, which is acceptable (though still a lot slower than the shell script that pipes into Perl, and I'm not sure why that is). "

I'm completely guessing but the overhead may be due to the IO::Uncompress::Bunzip2 module. You could avoid using that module by setting up the same pipe but from within the Perl script (rather than piping to that script).

I put exactly the same data I used previously into a text file (just a copy and paste):

$ cat > pm_1196493_paragraph_mode_test_data.txt
Block1 Line1
...
Block4 Line3
^D

I then modified the start of my previous example code, so it now looks like this:

#!/usr/bin/env perl -l

use strict;
use warnings;
use autodie;

my $filename = 'pm_1196493_paragraph_mode_test_data.txt';

open my $z, '-|', "cat $filename";

{
    local $/ = '';

    while (<$z>) {
        chomp;
        print '--- One Block ---';
        print;
    }
}

This produces exactly the same output as before. Obviously, you'll want to change 'cat' to '/usr/bin/bzcat' (and, of course, use *.bz2 instead of *.txt files). This solution will not be platform-independent: that may not matter to you. See open for more on the '-|', and closely related '|-', modes.

Also, note that I used the autodie pragma. If you want more control over handling I/O problems, you can hand-craft messages (e.g. open ... or die "..."), or use something like Try::Tiny.

[JediWombat]

Thanks again, Ken. I've built this code using your and Mario's responses:

   $/ = "";

   open my $fh, "-|", "/usr/bin/bzcat $file";

   while (<$fh>) {
       if (/uid=$mbnum/m) {
           print $_;
           last;
       }
   }

I've timed this version, and all others: this completed in 3.2 seconds, the previous version I built with your help took 8 seconds, and my original took 25.4! As I need to scan through three different LDIF's, that's a total of under 10 seconds, on average.

[marioroy]

Hi JediWombat,

The following provides a parallel demonstration to process all 3 LDIF files simultaneously. It requires MCE 1.830 minimally due to MCE::Relay failing from setting the input separator. That said, am posting the code before releasing 1.830 in order to have the perlmonks URL for the changelog.

Well, here is the code.

#!/usr/bin/perl

use strict;
use warnings;

# Requires MCE 1.830. Fixes MCE::Relay stalling from setting $/.

use MCE::Loop 1.830;

my $mbnum = $ARGV[0] or die "usage: $0 mbnum\n";

my @ldif_files = qw(
    /path/to/file1.ldif.bz2
    /path/to/file2.ldif.bz2
    /path/to/file3.ldif.bz2
);

MCE::Loop->init(
    max_workers => scalar @ldif_files,
    chunk_size  => 1,
    init_relay  => ''
);

mce_loop {
    # my ($mce, $chunk_ref, $chunk_id) = @_;
    # When chunk_size 1 is specified, $_ contains
    # the same value as $chunk_ref->[0].

    my ($file, $ret) = ($_, '');

    # Must localize $/ to not stall MCE, fixed in 1.830.
    # Localizing $/ is recommended, but fixed MCE if not.

    local $/ = "";

    open my $fh, "-|", "/usr/bin/bzcat $file"
        or warn "open error ($file): $!";

    if (defined fileno($fh)) {
        while (<$fh>) {
            if (/uid=$mbnum/m) {
                $ret  = "## $file\n";
                $ret .= $_;
                last;
            }
        }
        close $fh;
    }

    # Relay is beneficial for running a code block serially
    # and orderly. The init_relay option enables MCE::Relay.
    # All participating workers must call relay. Here, workers
    # write to STDOUT orderly starting with chunk_id 1.

    MCE::relay { print $ret };

} \@ldif_files;

MCE::Loop->finish;

MCE 1.830 will be released on CPAN no latter than Monday, the 7th of August. In the meantime, the MCE Github repository is current.

Regards, Mario

[marioroy]

Another possibility is sending the result to the manager process via MCE->gather. MCE::Candy provides an ordered output iterator.

Unlike the previous demonstration, this one doesn't require MCE 1.830 minimally.

#!/usr/bin/perl

use strict;
use warnings;

use MCE::Loop;
use MCE::Candy;

my $mbnum = $ARGV[0] or die "usage: $0 mbnum\n";

my @ldif_files = qw(
    /path/to/file1.ldif.bz2
    /path/to/file2.ldif.bz2
    /path/to/file3.ldif.bz2
);

MCE::Loop->init(
    chunk_size => 1, max_workers => scalar @ldif_files,
    gather => MCE::Candy::out_iter_fh(\*STDOUT)
);

mce_loop {
    my ($mce, $chunk_ref, $chunk_id) = @_;
    my ($file, $ret) = ($chunk_ref->[0], '');

    # Must localize $/ to not stall MCE, fixed in 1.830.
    # Localizing $/ is recommended, but fixed MCE if not.

    local $/ = "";

    open my $fh, "-|", "/usr/bin/bzcat $file"
        or warn "open error ($file): $!";

    if (defined fileno($fh)) {
        while (<$fh>) {
            if (/uid=$mbnum/m) {
                $ret  = "## $file\n";
                $ret .= $_;
                last;
            }
        }
        close $fh;
    }

    # The out_iter_fh iterator wants the chunk_id value.
    # Thus, all participating workers must call gather once only.
    # The manager process outputs the value for chunk_id 1 first,
    # then chunk_id 2, et cetera.

    MCE->gather($chunk_id, $ret);

} \@ldif_files;

MCE::Loop->finish;

Regards, Mario

[marioroy]

Yet another demonstration to be sure MCE::Hobo and MCE::Shared are not impacted when modifying the record separtor. Not localizing the record separator works too. Workers store the result into a shared array.

#!/usr/bin/perl

use strict;
use warnings;

use MCE::Hobo;
use MCE::Shared;

my $mbnum = $ARGV[0] or die "usage: $0 mbnum\n";

my @ldif_files = qw(
    /path/to/file1.ldif.bz2
    /path/to/file2.ldif.bz2
    /path/to/file3.ldif.bz2
);

my $ret = MCE::Shared->array();

for my $idx (0 .. $#ldif_files) {
    my $file = $ldif_files[$idx];

    mce_async {
        local $/ = ""; $ret->set($idx, "");

        open my $fh, "-|", "/usr/bin/bzcat $file"
            or warn "open error ($file): $!";

        if (defined fileno($fh)) {
            while (<$fh>) {
                if (/uid=$mbnum/m) {
                    $ret->append($idx, $_);
                    last;
                }
            }
            close $fh;
        }
    };
}

$_->join for MCE::Hobo->list;   # or MCE::Hobo->waitall;

print join('', $ret->values) if $ret->len;

Regards, Mario