Re: question about running a system and a perl function background

[BrowserUk]

You want to be able to:

1. Run several external commands concurrently.
2. Check the output from those commands for some keyword(s).
3. (Optionally?) Have the outputs end up in files. (or is this just so you can search the output later?).
4. Have the script that starts the concurrent processes know:
   1. when the tasks completed.
   2. Whether the output contained the keyword(s).

threads and the 'piped' version of open (see perlopentut) makes this kind of concurrency easy.

#! perl -slw
use strict;
use threads qw[ async ];
use threads::shared;

sub runAndCheck {
    my( $cmd, $lookFor, $file, $checkRef, $doneRef ) = @_;
    open OUT, '>', $file or die "$file : $!";
    open CMD, "$cmd |" or die "$cmd : $!";
    while ( <CMD> ) {
        $$checkRef = 1 if $_ =~ $lookFor;
        print OUT;
    }
    close CMD;
    close OUT;
    $$doneRef = 1;
}

my @cmds = map{ "dir $_" } qw[ c:\ P:\test r:\ d:\ ];
my @checks : shared = ( 0 ) x @cmds;
my @dones  : shared = ( 0 ) x @cmds;
my $lookFor = 'junk.htm';

async{ 
    runAndCheck( 
        $cmds[ $_ ],
        $lookFor,
        "test$_.out",
        \$checks[ $_ ],
        \$dones[ $_ ]
    );
}->detach for 0 .. $#cmds;

while( grep( $_, @dones ) < @cmds ) {
    sleep 1;
    for ( 0 .. $#cmds ) {
        if( $dones[ $_ ] == 1 ) {
            printf "'$cmds[ $_ ]' completed; '$lookFor' ";
            print $checks[ $_ ] ? 'was found' : 'was not found';
            $dones[ $_ ] = 2;
        }
    }
}

__END__
P:\test>412014
'dir c:\' completed; 'junk.htm' was not found
'dir P:\test' completed; 'junk.htm' was found
'dir r:\' completed; 'junk.htm' was not found
'dir d:\' completed; 'junk.htm' was not found

P:\test>412014
'dir c:\' completed; 'junk.htm' was not found
'dir r:\' completed; 'junk.htm' was not found
'dir d:\' completed; 'junk.htm' was not found
'dir P:\/s' completed; 'junk.htm' was found

[Anonymous Monk]

Exactly. That is exactly what I was trying to do. I tried your script, and get the following error:

This Perl hasn't been configured and built properly for the threads mo
+dule to work. (The 'useitthreads' configuration option hasn't been us
+ed.)

Having threads support requires all of Perl and all of the XS modules 
+in the Perl installation to be rebuit, it is not just a question of a
+dding the threads module. (In other words, threaded and non-threaded 
+Perls are binary compatible.)

If you want the use of the threads module, please contact the people w
+ho built your Perl. 

Cannot continue, aborting.
BEGIN failed--compilation aborted at /usr/lib/perl5/5.8.0/cygwin-multi
+-64int/threads.pm line 28.

Any advice? Thanks.

[BrowserUk]

Yes. Install a newer version of perl that is configured and built with threads enabled. I would recommend that you get 5.8.4 or 5.8.5.

[Anonymous Monk]

Hi,

I got threads working in my new perl. I tried your suggestion and had a question.

If I was searching for certain keywords i.e.

@lookFor = ( "junk.htm", "serious.txt" )

And when I find "serious.txt", Is there a way I can immediatly stop all the other threads from completing?

Thank you.

[BrowserUk]

And when I find "serious.txt", Is there a way I can immediatly stop all the other threads from completing?

By that I assume you to mean that you want searching to cease once you have found something that matches all your targets.

NOTE: As coded, the things to search for are themselves used as regex. As such, you need to escape any meta characters on the command line. It is also case sensitive. I consider these a bonus for the application shown, but you may wan't to change one or the other.

#! perl -slw
use strict;
use threads qw[ async ];
use threads::shared;

our $DRIVES ||= 'C';

die "Nothing to search for" unless $ARGV[ 0 ];

sub runAndCheck {
    my( $cmd, $foundRef, $doneRef, $lookfor ) = @_;
    my $pid = open CMD, "$cmd |" or die "$cmd : $!";
    while ( my $line = <CMD> ) {
        chomp $line;
        last if keys %$lookfor == grep defined, values %$lookfor;
        $line =~ $_ and $lookfor->{ $_ } = $line for keys %{ $lookfor 
+};
    }
    close CMD;
    kill 1, $pid;
    $$doneRef = 1;
}

my @cmds = map{ "attrib /s $_:\\* " } split '', $DRIVES;
my @found  : shared;
my @dones  : shared = ( 0 ) x @cmds;
my %lookfor: shared; @lookfor{ @ARGV } = ();

async{ 
    runAndCheck( 
        $cmds[ $_ ],
        \@found,
        \$dones[ $_ ],
        \%lookfor        
    );
}->detach for 0 .. $#cmds;

sleep 1 while grep( $_, @dones ) < @cmds;

if( grep defined, %lookfor ) {
    printf "%20s : %s\n",
        $_, defined $lookfor{ $_ } 
        ? " was found at '$lookfor{ $_ }'"
        : " was not found."
            for keys %lookfor;
}
else {
    print "None of @ARGV were found";
}

Some examples runs:

Read more... (2 kB)

[Anonymous Monk]

Hello

Thanks for the reply. Actually, if I find 'serious.txt' only, I want the other threads to stop immediatly. If I find 'junk.htm', I want the other threads to continue and complete. Please advice.

Also I had another question ... I seem to get "Scalar Leaked" error after my third thread is started...and hence I don't see my fourth thread start...what could I be doing wrong?

Thank you.

[BrowserUk]

Actually, if I find 'serious.txt' only, I want the other threads to stop immediatly. If I find 'junk.htm', I want the other threads to continue and complete. Please advice.

Advice? Sure. Play with the code. Understand how it works and under what conditions the threads stop early. Then devise a way of conveying which of the looked for terms are stoppers and which are carryons. Then add logic to perform that function.

I seem to get "Scalar Leaked" error ...what could I be doing wrong?

Which version of Perl are you using?

That message used to come up a lot in versions 5.8.0-5.8.2. With 5.8.3 is became much less common but still happened under some circumstances--mostly avoidable if you knew how though.

I use 5.8.4, and I do not recall having ever seen that particular error with it.

I recommend 5.8.4 over 5.8.5 for threaded work simply because whilst I have 5.8.5, I have done very little with it.

I have not yet downloaded 5.8.6 and probably will not until AS make their version available, if they ever do. I took a quick look at the delta document and nothing leapt off the page as a "must have".

Certainly nothing that warrents me going through the process of downloading and building my own version. Followed by all the pain of downloading/ building/ installing all the extra stuff that AS add to their packages. Followed by working out all the stuff I've added to my installation through any of the half dozen different ways I install different packages from any of a dozen or so different sources.

[Anonymous Monk]

Hi,

Thanks for the reply. I am using perl version 5.8.4 which you had suggested earlier.

I have been playing with the code...but could not understand:

        last if keys %$lookfor == grep defined, values %$lookfor;
        $line =~ $_ and $lookfor->{ $_ } = $line for keys %{ $lookfor 
+};

I understand it means that for keys %{ $lookFor }, if $line =~ $_, then $lookFor->{$_} = $line...but was not sure what if keys %$lookfor == grep defined, values %$lookfor meant.

Also, had a little trouble understanding what the following means... (what does < do ?) :

sleep 1 while grep( $_, @dones ) < @cmds;

I understood that when the keywords are found, kill 1, $pid immediatly kills the pid for that particular thread, but does it also stop the other 3 threads that are still going on?

Thank you.