Fork and WWW::Mechanize: Can agent be shared?

[cormanaz]

Good day monks. I am trying to get the hang of using multiple processes with WWW::Mechanize. I think I've just about got it figured out except for one thing. Can different processes share the same mech object? In other words can I

use Proc::Queue size => 3;
my $mech = WWW::Mechanize->new(timeout => 90);
while (@urllist) {
    $pid = fork;
    if(defined ($pid) and $pid==0) {
        $mech->get($url);
        print "$mech->content()->as_HTML();
        exit(0);
    }
    1 while waitpid(-1, WNOHANG)>0; # reaps childs
}

or do I have to

use Proc::Queue size => 3;
while (@urllist) {
    $pid = fork;
    if(defined ($pid) and $pid==0) {
    my $mech = WWW::Mechanize->new(timeout => 90);
        $mech->get($url);
        print "$mech->content()->as_HTML();
        exit(0);
    }
    1 while waitpid(-1, WNOHANG)>0; # reaps childs
}

?

Many thanks...

Steve

[andyford]

You can do it the first way. But after the fork it's no longer the same object, it's a copy, so you can't share info.

[shmem]

As said before, the first way should work - but I really wouldn't place the waitpid inside the loop, but move that into a $SIG{CHLD} handler.

[perrin]

It can be shared, because it won't actually be shared, if you see what I mean. The only thing you have to worry about with shared objects and forking is forking after opening a socket or file, which I don't think will happen here.

[ikegami]

You'll probably find Parallel::ForkManager useful.

use Parallel::ForkManager qw( );

use constant MAX_CHILDREN => 3;

{
   my $mech = WWW::Mechanize->new(timeout => 90);

   my @urllist = ...;

   my $pm = Parallel::ForkManager->new(MAX_CHILDREN);
   foreach my $url (@urllist) {
      # Forks and returns the pid for the child.
      my $pid = $pm->start() and next; 

      $mech->get($url);
      print $mech->content()->as_HTML();

      # Exit child.
      $pm->finish();
   }
}

[TOD]

if you take a shared memory segment you can store a serialized form of the object you want to be common to all processes on it. it must be serialized, because you can't store references on a shmem segment.

but basically i would prefer threads instead of forking.

[jdrago_999]

If you need to share data between multiple processes, take a look at the forks and forks::shared modules (as well as the threads and threads::shared modules).

The semantics are different than what you're doing now, and there is some extra overhead, but I've been using both threads and forks for a couple years now with no problem, in production environments no less.

[tphyahoo]

You may find Using DBM::Deep and Parallel::ForkManager for a generalized parallel hashmap function builder (followup to "reducing pain of parallelization with FP") and/or Using functional programming to reduce the pain of parallel-execution programming (with threads, forks, or name your poison) to be helpful.

I admit however, that after playing around with different ways to get WWW::Mechanize behaving in a parallel way, eventually I gave up, shrugged, and kept doing things serially. (Parallel in my case turned out to be a bit of a premature optimization, other features were more important.)