Beefy Boxes and Bandwidth Generously Provided by pair Networks Bob
Perl-Sensitive Sunglasses
 
PerlMonks  

Re: question about running a system and a perl function background

by BrowserUk (Pope)
on Dec 03, 2004 at 04:50 UTC ( #412041=note: print w/ replies, xml ) Need Help??


in reply to question about running a system and a perl function background

You want to be able to:

  1. Run several external commands concurrently.
  2. Check the output from those commands for some keyword(s).
  3. (Optionally?) Have the outputs end up in files. (or is this just so you can search the output later?).
  4. Have the script that starts the concurrent processes know:
    1. when the tasks completed.
    2. Whether the output contained the keyword(s).

threads and the 'piped' version of open (see perlopentut) makes this kind of concurrency easy.

#! perl -slw use strict; use threads qw[ async ]; use threads::shared; sub runAndCheck { my( $cmd, $lookFor, $file, $checkRef, $doneRef ) = @_; open OUT, '>', $file or die "$file : $!"; open CMD, "$cmd |" or die "$cmd : $!"; while ( <CMD> ) { $$checkRef = 1 if $_ =~ $lookFor; print OUT; } close CMD; close OUT; $$doneRef = 1; } my @cmds = map{ "dir $_" } qw[ c:\ P:\test r:\ d:\ ]; my @checks : shared = ( 0 ) x @cmds; my @dones : shared = ( 0 ) x @cmds; my $lookFor = 'junk.htm'; async{ runAndCheck( $cmds[ $_ ], $lookFor, "test$_.out", \$checks[ $_ ], \$dones[ $_ ] ); }->detach for 0 .. $#cmds; while( grep( $_, @dones ) < @cmds ) { sleep 1; for ( 0 .. $#cmds ) { if( $dones[ $_ ] == 1 ) { printf "'$cmds[ $_ ]' completed; '$lookFor' "; print $checks[ $_ ] ? 'was found' : 'was not found'; $dones[ $_ ] = 2; } } } __END__ P:\test>412014 'dir c:\' completed; 'junk.htm' was not found 'dir P:\test' completed; 'junk.htm' was found 'dir r:\' completed; 'junk.htm' was not found 'dir d:\' completed; 'junk.htm' was not found P:\test>412014 'dir c:\' completed; 'junk.htm' was not found 'dir r:\' completed; 'junk.htm' was not found 'dir d:\' completed; 'junk.htm' was not found 'dir P:\/s' completed; 'junk.htm' was found

Examine what is said, not who speaks.
"But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
"Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
"Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon


Comment on Re: question about running a system and a perl function background
Download Code
Re^2: question about running a system and a perl function background
by Anonymous Monk on Dec 03, 2004 at 17:38 UTC
    Exactly. That is exactly what I was trying to do. I tried your script, and get the following error:

    This Perl hasn't been configured and built properly for the threads mo +dule to work. (The 'useitthreads' configuration option hasn't been us +ed.) Having threads support requires all of Perl and all of the XS modules +in the Perl installation to be rebuit, it is not just a question of a +dding the threads module. (In other words, threaded and non-threaded +Perls are binary compatible.) If you want the use of the threads module, please contact the people w +ho built your Perl. Cannot continue, aborting. BEGIN failed--compilation aborted at /usr/lib/perl5/5.8.0/cygwin-multi +-64int/threads.pm line 28.
    Any advice? Thanks.

      Yes. Install a newer version of perl that is configured and built with threads enabled. I would recommend that you get 5.8.4 or 5.8.5.


      Examine what is said, not who speaks.
      "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
      "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
      "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Re^2: question about running a system and a perl function background
by Anonymous Monk on Dec 15, 2004 at 03:21 UTC
    Hi,

    I got threads working in my new perl. I tried your suggestion and had a question.

    If I was searching for certain keywords i.e.

    @lookFor = ( "junk.htm", "serious.txt" )

    And when I find "serious.txt", Is there a way I can immediatly stop all the other threads from completing?

    Thank you.

      And when I find "serious.txt", Is there a way I can immediatly stop all the other threads from completing?

      By that I assume you to mean that you want searching to cease once you have found something that matches all your targets.

      NOTE: As coded, the things to search for are themselves used as regex. As such, you need to escape any meta characters on the command line. It is also case sensitive. I consider these a bonus for the application shown, but you may wan't to change one or the other.

      #! perl -slw use strict; use threads qw[ async ]; use threads::shared; our $DRIVES ||= 'C'; die "Nothing to search for" unless $ARGV[ 0 ]; sub runAndCheck { my( $cmd, $foundRef, $doneRef, $lookfor ) = @_; my $pid = open CMD, "$cmd |" or die "$cmd : $!"; while ( my $line = <CMD> ) { chomp $line; last if keys %$lookfor == grep defined, values %$lookfor; $line =~ $_ and $lookfor->{ $_ } = $line for keys %{ $lookfor +}; } close CMD; kill 1, $pid; $$doneRef = 1; } my @cmds = map{ "attrib /s $_:\\* " } split '', $DRIVES; my @found : shared; my @dones : shared = ( 0 ) x @cmds; my %lookfor: shared; @lookfor{ @ARGV } = (); async{ runAndCheck( $cmds[ $_ ], \@found, \$dones[ $_ ], \%lookfor ); }->detach for 0 .. $#cmds; sleep 1 while grep( $_, @dones ) < @cmds; if( grep defined, %lookfor ) { printf "%20s : %s\n", $_, defined $lookfor{ $_ } ? " was found at '$lookfor{ $_ }'" : " was not found." for keys %lookfor; } else { print "None of @ARGV were found"; }

      Some examples runs:


      Examine what is said, not who speaks.        The end of an era!
      "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
      "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
      "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
Re^2: question about running a system and a perl function background
by Anonymous Monk on Dec 15, 2004 at 14:37 UTC
    Hello

    Thanks for the reply. Actually, if I find 'serious.txt' only, I want the other threads to stop immediatly. If I find 'junk.htm', I want the other threads to continue and complete. Please advice.

    Also I had another question ... I seem to get "Scalar Leaked" error after my third thread is started...and hence I don't see my fourth thread start...what could I be doing wrong?

    Thank you.

      Actually, if I find 'serious.txt' only, I want the other threads to stop immediatly. If I find 'junk.htm', I want the other threads to continue and complete. Please advice.

      Advice? Sure. Play with the code. Understand how it works and under what conditions the threads stop early. Then devise a way of conveying which of the looked for terms are stoppers and which are carryons. Then add logic to perform that function.

      I seem to get "Scalar Leaked" error ...what could I be doing wrong?

      Which version of Perl are you using?

      That message used to come up a lot in versions 5.8.0-5.8.2. With 5.8.3 is became much less common but still happened under some circumstances--mostly avoidable if you knew how though.

      I use 5.8.4, and I do not recall having ever seen that particular error with it.

      I recommend 5.8.4 over 5.8.5 for threaded work simply because whilst I have 5.8.5, I have done very little with it.

      I have not yet downloaded 5.8.6 and probably will not until AS make their version available, if they ever do. I took a quick look at the delta document and nothing leapt off the page as a "must have".

      Certainly nothing that warrents me going through the process of downloading and building my own version. Followed by all the pain of downloading/ building/ installing all the extra stuff that AS add to their packages. Followed by working out all the stuff I've added to my installation through any of the half dozen different ways I install different packages from any of a dozen or so different sources.


      Examine what is said, not who speaks.        The end of an era!
      "But you should never overestimate the ingenuity of the sceptics to come up with a counter-argument." -Myles Allen
      "Think for yourself!" - Abigail        "Time is a poor substitute for thought"--theorbtwo         "Efficiency is intelligent laziness." -David Dunham
      "Memory, processor, disk in that order on the hardware side. Algorithm, algorithm, algorithm on the code side." - tachyon
        Hi,

        Thanks for the reply. I am using perl version 5.8.4 which you had suggested earlier.

        I have been playing with the code...but could not understand:

        last if keys %$lookfor == grep defined, values %$lookfor; $line =~ $_ and $lookfor->{ $_ } = $line for keys %{ $lookfor +};

        I understand it means that for keys %{ $lookFor }, if $line =~ $_, then $lookFor->{$_} = $line...but was not sure what if keys %$lookfor == grep defined, values %$lookfor meant.

        Also, had a little trouble understanding what the following means... (what does < do ?) :

        sleep 1 while grep( $_, @dones ) < @cmds;

        I understood that when the keywords are found, kill 1, $pid immediatly kills the pid for that particular thread, but does it also stop the other 3 threads that are still going on?

        Thank you.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://412041]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (7)
As of 2014-04-19 20:58 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (483 votes), past polls