Working in parallel

[mpapec]

I'm working with multiple Net::Telnet instances and would like to simultaneously collect data from them, storing data in some common hash. Is there module which could help me in efficient way with this task? TIA

[BrowserUk]

Something like this?

#! perl -slw
use strict;
use threads;
use threads::shared;
use Net::Telnet;

my %data :shared;

sub thread {
    my( $machine, $user, $pass, $cmd ) = @_;
    $t = new Net::Telnet (
        Timeout => 10,
        Prompt => '>'
    );
    $t->open( $machine );
    $t->login( $user, $pass );
    my @lines :shared = $t->cmd( $cmd );
    $t->close;
    lock %data;
    $data{ $machine } = \@lines;
    return;
}

# AoA each sub array containing machineId, username, password and comm
+and to run
my @credentials = ...;

my @threads = map threads->create( \&thread, @$_ ), @credentials;

$_->join for @threads;

## Do something useful with the data gathered in %data.

[sundialsvc4]

“Show off...”   ;-)

Naww, seriously... that's what I love about Perl.   You can describe this big, hairy-looking task, and then “just whip it out” in a few dozen lines.

Now, I’ll happily spend the next hour or so poring over what you just wrote.

[BrowserUk]

that's what I love about Perl. You can describe this big, hairy-looking task, and then “just whip it out” in a few dozen lines.

Ditto. That is as good a description of what is good about Perl as I seen in a long time.

It's just a shame that so many people want to turn it into Java; in the interests of "modernity".

[Anonymous Monk]

Tnx, it does what I want! ps. I guess $_->join for @threads; waits for all threads to finish?

[BrowserUk]

I guess $_->join for @threads; waits for all threads to finish?

Correct. Though I have to say that subs::parallel might be even easier for your purpose.

[BioLion]

Check out CPAN - there are modules for helping run and manage children :

1. Parallel::ForkManager (one of the simplest)
2. Parallel::Forker (allows a lot more fine grained control)
3. There are a lot of others too, which may suit your needs more or less, so take time to browse...

There are also a number of IPC modules :

1. perlipc has a lot more details
2. e.g. IPC::Shareable
I have had good results with this one, but there are a lot others and I might be a bit out of date!

Hope this helps - let us know if you get a bit further and still have problems/questions!

[Anonymous Monk]

Thank you for your quick reply! I've looked into Parallell::ForkManager and IPC::Shareable but unfortunately they use Storable to exchange data (even worse, forkmanager writes Storable data onto disk :)) Then I've found subs::parallel which looks promising, and AFAIK doesn't use serialization.

  use Data::Dumper;
  use subs::parallel;

  my $baz = parallelize {
    my $h = {foo => 3};
    return $h;
  }; # returns immediately


  $baz and 1; # blocks
  print Dumper $baz;

[BrowserUk]

subs::parallel is *really* cute. It's been around since 2005, and yet I've never seen it before.

So amazingly simple and yet so very powerful. Thanks for bringing to the community's attention.

[zentara]

See Simple threaded chat server. You can share filhandles across threads thru the fileno's. That make it easy to use some sort of select or filehandle-watch in the main thread, by sharing the filhandles thru threads:shared variables.

[james2vegas]

Since Net::Telnet is a IO::Socket subclass, you can use it as the handle to the connection with your server. You can then use an event, select or poll-based interface to wait for data on your multiple connections, f.e. Event, EV, AnyEvent::Handle, POE::Wheel::ReadWrite, IO::Select, IO::Poll, select and poll.

This way you can use one process to collect all your data, avoiding forking, threading or IPC.

[sundialsvc4]

That brings up a very important point:   many server programs need to handle “many, even hundreds, of input channels,” but a simple socket-select design often works beautifully for them, with no need for threads.

Also, if the design does start getting complicated, a little bell should go off in your head:   “this has probably been done a thousand times before, and it's got to be on CPAN somewhere.”   So, before you go far down the primrose-path of coding something new, you stop writing and start looking.   Works every time.   You can find everything from well-tested component pieces, to battle-hardened complete frameworks.

[Anonymous Monk]

Could you post a simple example?

[james2vegas]

Sure, I'll use IO::Select for simplicity's sake, though I normally would use AnyEvent::Handle.


Read more... (1506 Bytes)