Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Need help with Perl multi threading

by ashok.g (Beadle)
on Jan 25, 2012 at 18:54 UTC ( #949944=perlquestion: print w/ replies, xml ) Need Help??
ashok.g has asked for the wisdom of the Perl Monks concerning the following question:

Hi Folks,

I need some help regarding Perl multi threading.

Can I have the following functionality using multi threads in Perl?

1. Main script calls a sub routine My_1st(), where My_list() requires two or more parameters.

2. My_list calls other sub routine My_1st_inside(), where this My_1st_inside() requires two or more parameters to it and returns a value to My_1st parent sub routine.

3. My_1st() gets the values returned by My_1st_inside() and prints some results.

I am able to achieve this without threads and its working fine except that it is taking much time in a foreach loop. So, I would like to add threads to this.

I tried this but resulting with errors. Is this something that can be achievable using Perl threads?

Any help is appreciated!!

Thanks,
Ashok.

Comment on Need help with Perl multi threading
Re: Need help with Perl multi threading
by Corion (Pope) on Jan 25, 2012 at 19:03 UTC

    Please show the code and data that does "not work" for you. Please make it self-contained and about 20 to 25 lines long. Thank you.

      Hi Corion,

      Thanks for your reply. Main code:
      #!/usr/bin/perl use threads; use strict; use warnings; use Net::SSH::Expect; require 'total.pl'; my @servers = ('192.168.0.18','192.168.0.19','192.168.0.20'); my @process = ('testing','b','c'); my @threads; my $output; my $res; foreach my $host ( @servers ) { my $png = `ping -c1 $host`; if ( $png =~ /64 bytes/ ) { my $user= 'netcool'; my $ssh = &SSH_Reference($host,$user); $ssh->run_ssh(); eval {$res=$ssh->read_all(); }; if ( !$@ && $res !~ /\(yes\/no\)/ && $res !~ m/password/ ) { print "\nLogged in to $host\n"; foreach my $proc ( @process ) { push (@threads, threads->new(\&Process_Output, $ss +h, $proc )); while ( my $thread = shift @threads ) { $output=$thread->join(); #print @output; if ( $output =~ m/$proc/ ) { print "\t$proc is runn +ing\n"; } else { print "\t$proc is not +running\n"; } } } $ssh->close(); } else { print "Unable to SSH to $host\n"; } } else { print "\nUnable to ping to Host: $host\n"; } }
      total.pl:
      use Config::General; use Tie::IxHash; use Net::SNMP; sub SSH_Reference() { ($host,$user)=@_; #print "$host---$user--\n"; #($host,$user)= @_; $ssh = Net::SSH::Expect->new ( host => $host, user => $user, timeout=>10, raw_pty => 1 ); return $ssh; } sub Process_Output() { $ssh = shift; $process = shift; #( $ssh, $process ) = @_; $output; $ssh->eat($ssh->peek(0)); $ssh->send("ps -ef | grep $process | grep -v 'grep'"); while ( defined ($line = $ssh->read_line(2)) ) { next if ( $line =~ m/grep/ ); $output .= "$line"; } $ssh->eat($ssh->peek(0)); return $output; } 1;
      And the output is:
      perl some.pl Logged in to 192.168.0.18 Use of uninitialized value in pattern match (m//) at some.pl line 33. testing is not running Thread 2 terminated abnormally: SSHConnectionAborted at total.pl line +25 Use of uninitialized value in pattern match (m//) at some.pl line 33. b is not running Thread 3 terminated abnormally: SSHConnectionAborted at total.pl line +25 Use of uninitialized value in pattern match (m//) at some.pl line 33. c is not running Logged in to 192.168.0.19 Segmentation fault
      Please suggest

      Thanks,
      Ashok

        The first, most obvious problem is that you did not use warnings; use strict;

        I see undeclared variables and other weird stuff. What's "$output;" all by itself in the middle of the function supposed to do anyways?

        I suspect your main problem is that you're trying to use the same instance of the SSH connection in all threads.  Creating a new thread doesn't automagically also create a new independent duplicate of the connection (socket, server-side process, and all), so the SSH will probably get mildly confused when multiple clients are talking simultaneously on the same line...

        Sorry to be rude, but you really don't seem to have a clue what you are doing.

        This is a completely redundant use of threading:

        push (@threads, threads->new(\&Process_Output, $ssh, $proc )); while ( $thread = shift @threads ) { @output=$thread->join(); }

        You create a thread and push its handle onto @threads. You then immediately shift that handle off of the array and join it.

        1. There is no point in using an array here, because it will only ever contain at most one handle.
        2. Which makes using the while loop redundant because it will only ever iterate once.

          It is exactly equivalent to:

          $thread = threads->new(\&Process_Output, $ssh, $proc ); if( $thread ) { @output = $thread->join(); }

          Which is equivalent to :

          $thread = threads->new(\&Process_Output, $ssh, $proc ); @output = $thread->join();

          Which is exactly equivalent to this:

          @output = threads->new(\&Process_Output, $ssh, $proc )->join();

          Which is exactly equivalent to:

          @output = Process_Output( $ssh, $proc );

          Except that the latter will work. (Of course, it won't speed things up. But neither will what you have now.) But it will work.

        The reason your code is crashing is because you are creating an instance of Net::SSH::Expect in one thread, and trying to use it in another.

        Why? Why not create the object in the thread where you are going to use it?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Re: Need help with Perl multi threading
by sundialsvc4 (Monsignor) on Jan 26, 2012 at 15:02 UTC
    I am able to achieve this without threads and its working fine except that it is taking much time in a foreach loop.  So, I would like to add threads to this.

    I would be hard-pressed to be convinced that “adding threads to” such an algorithm that is right now “working fine(!)” will actually make it either more reliable, or faster.   Threading always divides time, while adding considerable complication.   So, all kinds of Danger, Will Robinson! alarm bells are going off in my head.   What is the procedure that is being performed by this loop that is taking too much time?   What is the source of the delay?

    Don’t “diddle” code to make it faster: find a better algorithm.
    – Kernighan and Plauger; The Elements of Programming Style.

      I would be hard-pressed to be convinced that “adding threads to” such an algorithm that is right now “working fine(!)” will actually make it either more reliable, or faster.

      Perhaps if you relied less on your apparently limited imagination and tried using the stuff you feel obliged to talk about, you might talk less crap.

      Threading always divides time, while adding considerable complication.

      What!? What on earth are you taking when you dream this stuff up? I mean really man, did you do no simple math at school:

      One man digs a ditch and it takes 8 hours. Two men dig another identical ditch and it takes 4.

      ... or were you simply to busy bullshitting about how you'd seen and done it all in pre-school, to take any notice.

      So, all kinds of Danger, Will Robinson! alarm bells are going off in my head.

      Geez! You're like a kid that was told masturbation would make him go blind and has refused wash his winkle ever since.

      A simple example of threading at work:

      Fetching 100 urls serially:

      #! perl -slw use strict; use LWP::Simple; my $start = time; while( <> ) { chomp; eval { local $SIG{ ALRM } = sub { die 'timeout' }; alarm 10; my($content_type, $document_length, $modified_time, $expires, +$server) = head( "http://$_" ) or warn "Failed to HEAD $_: $!\n"; }; } printf "\nRunning HEAD on $. urls serially took %d seconds\n", time() - $start; __END__ C:\test>junk57 urls.list.small Failed to HEAD www.asuscom.de: Bad file descriptor Failed to HEAD www.belkin.com: Bad file descriptor Failed to HEAD www.logical-approach.com: Bad file descriptor Running HEAD on 100 urls serially took 102 seconds

      Fetching those same urls in parallel:

      #! perl -slw use strict; use threads; use LWP::Simple; my $start = time; my @threads; while( <> ) { chomp; push @threads, async{ eval { local $SIG{ ALRM } = sub { die 'timeout' }; alarm 10; my( $content_type, $document_length, $modified_time, $expi +res, $server ) = head( "http://$_" ) or warn "Failed to HEAD $_: $!\n"; }; }; } $_->join for @threads; printf "\nRunning HEAD on $. urls using threading took %d seconds\n", time() - $start; __END__ C:\test>junk58 urls.list.small Failed to HEAD www.asuscom.de: Failed to HEAD www.belkin.com: Failed to HEAD www.logical-approach.com: Running HEAD on 100 urls using threading took 19 seconds

      5 times faster -- and look at how much more complicated it is. Four whole extra lines!


      With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

      The start of some sanity?

      I would be hard-pressed to be convinced that “adding threads to” such an algorithm that is right now “working fine(!)” will actually make it either more reliable, or faster.

      In a case like this, where the code performs network IO, it's easy to convince me that having multiple things block on IO simultaneously can be faster than code which has to block on IO serially.

      I use parallelism all the time in similar situations to great effect.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://949944]
Approved by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (8)
As of 2014-07-22 09:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (109 votes), past polls