question about running a system and a perl function background

[Anonymous Monk]

I would like to run a system process in the background. The system process redirects output to a file. Immediatly after that process completes, I want to run a perl function that will check for a particular string in the output file.

I am running 4 such things simultaneously.

system 1, ( "some process 1" );
system 1, ( "some process 2" );
system 1, ( "some process 3" );
system 1, ( "some process 4" );

But immediatly after each of the above process completes, I want to run my perl error checking function. Please suggest how I can run that.

I was thinking I might be able to do something like:

system 1, ( "some process 1; perl checkerror.pl" );
system 1, ( "some process 2; perl checkerror.pl" );
system 1, ( "some process 3; perl checkerror.pl" );
system 1, ( "some process 4; perl checkerror.pl" );

I have made the checkerror a function, so I don't want to make it a perl script all by itself. Any advice?

Also, is there a way I can know when all the 4 processes have completed?

Thanks

[BrowserUk]

You want to be able to:

1. Run several external commands concurrently.
2. Check the output from those commands for some keyword(s).
3. (Optionally?) Have the outputs end up in files. (or is this just so you can search the output later?).
4. Have the script that starts the concurrent processes know:
   1. when the tasks completed.
   2. Whether the output contained the keyword(s).

threads and the 'piped' version of open (see perlopentut) makes this kind of concurrency easy.

#! perl -slw
use strict;
use threads qw[ async ];
use threads::shared;

sub runAndCheck {
    my( $cmd, $lookFor, $file, $checkRef, $doneRef ) = @_;
    open OUT, '>', $file or die "$file : $!";
    open CMD, "$cmd |" or die "$cmd : $!";
    while ( <CMD> ) {
        $$checkRef = 1 if $_ =~ $lookFor;
        print OUT;
    }
    close CMD;
    close OUT;
    $$doneRef = 1;
}

my @cmds = map{ "dir $_" } qw[ c:\ P:\test r:\ d:\ ];
my @checks : shared = ( 0 ) x @cmds;
my @dones  : shared = ( 0 ) x @cmds;
my $lookFor = 'junk.htm';

async{ 
    runAndCheck( 
        $cmds[ $_ ],
        $lookFor,
        "test$_.out",
        \$checks[ $_ ],
        \$dones[ $_ ]
    );
}->detach for 0 .. $#cmds;

while( grep( $_, @dones ) < @cmds ) {
    sleep 1;
    for ( 0 .. $#cmds ) {
        if( $dones[ $_ ] == 1 ) {
            printf "'$cmds[ $_ ]' completed; '$lookFor' ";
            print $checks[ $_ ] ? 'was found' : 'was not found';
            $dones[ $_ ] = 2;
        }
    }
}

__END__
P:\test>412014
'dir c:\' completed; 'junk.htm' was not found
'dir P:\test' completed; 'junk.htm' was found
'dir r:\' completed; 'junk.htm' was not found
'dir d:\' completed; 'junk.htm' was not found

P:\test>412014
'dir c:\' completed; 'junk.htm' was not found
'dir r:\' completed; 'junk.htm' was not found
'dir d:\' completed; 'junk.htm' was not found
'dir P:\/s' completed; 'junk.htm' was found

[Anonymous Monk]

Exactly. That is exactly what I was trying to do. I tried your script, and get the following error:

This Perl hasn't been configured and built properly for the threads mo
+dule to work. (The 'useitthreads' configuration option hasn't been us
+ed.)

Having threads support requires all of Perl and all of the XS modules 
+in the Perl installation to be rebuit, it is not just a question of a
+dding the threads module. (In other words, threaded and non-threaded 
+Perls are binary compatible.)

If you want the use of the threads module, please contact the people w
+ho built your Perl. 

Cannot continue, aborting.
BEGIN failed--compilation aborted at /usr/lib/perl5/5.8.0/cygwin-multi
+-64int/threads.pm line 28.

Any advice? Thanks.

[BrowserUk]

Yes. Install a newer version of perl that is configured and built with threads enabled. I would recommend that you get 5.8.4 or 5.8.5.

[Anonymous Monk]

Hi,

I got threads working in my new perl. I tried your suggestion and had a question.

If I was searching for certain keywords i.e.

@lookFor = ( "junk.htm", "serious.txt" )

And when I find "serious.txt", Is there a way I can immediatly stop all the other threads from completing?

Thank you.

[BrowserUk]

And when I find "serious.txt", Is there a way I can immediatly stop all the other threads from completing?

By that I assume you to mean that you want searching to cease once you have found something that matches all your targets.

NOTE: As coded, the things to search for are themselves used as regex. As such, you need to escape any meta characters on the command line. It is also case sensitive. I consider these a bonus for the application shown, but you may wan't to change one or the other.

#! perl -slw
use strict;
use threads qw[ async ];
use threads::shared;

our $DRIVES ||= 'C';

die "Nothing to search for" unless $ARGV[ 0 ];

sub runAndCheck {
    my( $cmd, $foundRef, $doneRef, $lookfor ) = @_;
    my $pid = open CMD, "$cmd |" or die "$cmd : $!";
    while ( my $line = <CMD> ) {
        chomp $line;
        last if keys %$lookfor == grep defined, values %$lookfor;
        $line =~ $_ and $lookfor->{ $_ } = $line for keys %{ $lookfor 
+};
    }
    close CMD;
    kill 1, $pid;
    $$doneRef = 1;
}

my @cmds = map{ "attrib /s $_:\\* " } split '', $DRIVES;
my @found  : shared;
my @dones  : shared = ( 0 ) x @cmds;
my %lookfor: shared; @lookfor{ @ARGV } = ();

async{ 
    runAndCheck( 
        $cmds[ $_ ],
        \@found,
        \$dones[ $_ ],
        \%lookfor        
    );
}->detach for 0 .. $#cmds;

sleep 1 while grep( $_, @dones ) < @cmds;

if( grep defined, %lookfor ) {
    printf "%20s : %s\n",
        $_, defined $lookfor{ $_ } 
        ? " was found at '$lookfor{ $_ }'"
        : " was not found."
            for keys %lookfor;
}
else {
    print "None of @ARGV were found";
}

Some examples runs:

Read more... (2 kB)

[Anonymous Monk]

Hello

Thanks for the reply. Actually, if I find 'serious.txt' only, I want the other threads to stop immediatly. If I find 'junk.htm', I want the other threads to continue and complete. Please advice.

Also I had another question ... I seem to get "Scalar Leaked" error after my third thread is started...and hence I don't see my fourth thread start...what could I be doing wrong?

Thank you.

[BrowserUk]

Actually, if I find 'serious.txt' only, I want the other threads to stop immediatly. If I find 'junk.htm', I want the other threads to continue and complete. Please advice.

Advice? Sure. Play with the code. Understand how it works and under what conditions the threads stop early. Then devise a way of conveying which of the looked for terms are stoppers and which are carryons. Then add logic to perform that function.

I seem to get "Scalar Leaked" error ...what could I be doing wrong?

Which version of Perl are you using?

That message used to come up a lot in versions 5.8.0-5.8.2. With 5.8.3 is became much less common but still happened under some circumstances--mostly avoidable if you knew how though.

I use 5.8.4, and I do not recall having ever seen that particular error with it.

I recommend 5.8.4 over 5.8.5 for threaded work simply because whilst I have 5.8.5, I have done very little with it.

I have not yet downloaded 5.8.6 and probably will not until AS make their version available, if they ever do. I took a quick look at the delta document and nothing leapt off the page as a "must have".

Certainly nothing that warrents me going through the process of downloading and building my own version. Followed by all the pain of downloading/ building/ installing all the extra stuff that AS add to their packages. Followed by working out all the stuff I've added to my installation through any of the half dozen different ways I install different packages from any of a dozen or so different sources.

[graff]

It sounds like you want to use fork() to launch each of the four processes -- something like this (untested): update: Based on ikegami's more knowledgeable reply, the following is the sort of thing you'd do if you weren't using Windows; but if you replace the fork stuff with your original "system 1,...", I think the while loop using "wait" will still do what you want.

my %children;

for my $n ( 1..4 )
{
    my $child = fork;
    if ( !defined $child) { die "fork failed on $n: $!" }
    elsif ( $child == 0 )
    {    # this is what the child does:
        exec 'some', 'process', $n;
    }
    $children{$child} = $n;
}
while ( keys %children )
{
    my $reaped_pid = wait;
    if ( exists( $children{$reaped_pid} )) {
        checkerror_function( $children{$reaped_pid} );
        delete $children{$reaped_pid};
    }
    else {
        # if you get here, better study "perldoc perlipc"
    }
}

I would assume that the "checkerror" function needs to know which of the sub-processes has just finished, so that it knows which set of output to check. That's why I kept track of the loop counter when forking, and pass it to checkerror_function when reaping.

(update: fixed missing close-quote and added slight improvement to "die" message)

[ikegami]

system 1, is Windows specific fork+exec that works better than fork+exec on Windows. By switching to fork+exec, you've taken a step backwards. system 1, returns the child's pid, so I think your solution could be rewritten to use system 1,.

[graff]

system 1, is Windows specific fork+exec that works better than fork+exec on Windows.

Ah. I was actually wondering about that -- I had never seen "system 1," before and wasn't even sure whether it was a mistake. (I'm blissfully ignorant of most of the Windows-specific extensions to Perl.) Thanks.

[Anonymous Monk]

Hello,

Is  system 1, supposed to return the child's pid? It seems to return -1 when I tested it out. Is there any other windows command that might return the child's pid?

Thanks for the response.

[Anonymous Monk]

Hello graff,

I have been attempting to do the same thing using threads according to BrowserUK's solution. But due to some complicated issues with the system I am using, I am trying to attempt the same thing using fork and exec instead.

Is it possible to have some arrangement whereby the original process sets up a separate STDOUT/STDIN prior to forking each command and then uses select to read the output from the child processes back into the parent for further possessing?

BrowserUK also is interested in seeing a working example using fork and exec.

So after the process forks, can I read each line that the process outputs, see if that line contains a search word I am looking for and if it does, stop all other forks immediately.

So for example if I am forking three processes: Dir C:, Dir I:, Dir P:

Then if any of those contain the searchword “Desktop”, then I would li
+ke to immediately stop all the other forks.

If you could provide a working example, me and BrowserUK would learn a lot.

Thank you.

[graff]

If I understand right, you want a form of IPC such that some number of processes can run in parallel, and as soon as any one of them comes up with "the answer", that job and all the others should immediately stop, and the parent should proceed to do stuff with the answer.

In this sort of case, playing with the stdin/stdout of all the children, and making the parent read all output from all children, strikes me as being more complicated than necessary (unless there's really something in your setup that makes it essential for the parent to see/process all the input from all children, and/or makes it impossible for the children to write to a commonly accessible disk).

Anyway, if you're actually just talking about a case where a few children need to run the same sort of search in parallel to cover a few different spaces, just set up the children so that each one will write whatever information the parent needs into a file. The parent should set the file name in advance (one that does not exist), and pass it along to all the children; then, after starting the kids going, it should just loop, watching for the file to come into existence. As soon as the file is there, "bye-bye, kids!"

If there's any chance of a race condition among children (i.e. two or more could come up with "the answer"), and especially if writing the file is more than just "open, write a record, close", it will probably be best for each child to write to its own personal temp file; once that file is complete and closed, check for existence of the parent's designated file, and if it does not exist, rename the temp file to the designated file name.

If the child processes involve something like "dir" or "find", i.e. some general-purpose tool that doesn't provide a way to stop as soon as some special condition is met, you could write a wrapper script to serve as the child process for the fork/exec; the child wrapper just needs to do a pipeline open of the general-purpose tool, and do the right thing when the target output shows up (closing the pipe file handle stops the tool).

(Sorry about not providing a working example, but I hope it's simple enough to work out on your own.)

[Mr. Muskrat]

First you post question about running a system and a perl function background and then this reply with more details today. No problem. So why did you feel the need to post forking and monitoring processes and now fork and exec as well?

[ikegami]

; doesn't work in Windows. Not even from the prompt. Use && instead of ;, and it works. (Tested using system 1, ( "echo aaa > file1 && copy file1 file2 > nul" );).

Use wait to detect when a child terminates.

[hsinclai]

If you have to "go to a system call", you could do a simple way like below for each (by the way there's no error checking in there at all) - but it's highly dependent I would say on what process you'll be running out there and on its particular ways of possibly failing, that will more determine what you'd write.


use strict;

my $joboutputfile = "/tmp/joboutputfile.out";

my $stuff;
#run job
qx!ls -al /var > $joboutputfile!;
( -e $joboutputfile ) && &checkerror;
print $stuff;


sub checkerror
{
  return $stuff = qx!grep "spool" $joboutputfile!;
}

[Anonymous Monk]

In the above code, if the output file exists, then the checkerror function is called. But, is there a way I know when the process had been completed?

This is because I need to wait for all four processes to finish before starting another 4 more simulataneously. And everytime I run 4 processes, they dump their output to the same files everytime.

So,

system 1, ("some process 1 > output1.txt");
system 1, ("some process 2 > output2.txt");
system 1, ("some process 3 > output3.txt");
system 1, ("some process 4 > output4.txt");

Then when I know each is completed, I check for error in those files: 
+If process 1 is completed, I'll check for error in output1.txt etc. W
+hen all are completed, I call next set of 4 processes

system 1, ("some process 5 > output1.txt");
system 1, ("some process 6 > output2.txt");
system 1, ("some process 7 > output3.txt");
system 1, ("some process 8 > output4.txt");

As seen above, the output files remain the same

Thanks.