Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Fork and WWW::Mechanize: Can agent be shared?

by cormanaz (Chaplain)
on Feb 18, 2007 at 17:13 UTC ( #600716=perlquestion: print w/replies, xml ) Need Help??

cormanaz has asked for the wisdom of the Perl Monks concerning the following question:

Good day monks. I am trying to get the hang of using multiple processes with WWW::Mechanize. I think I've just about got it figured out except for one thing. Can different processes share the same mech object? In other words can I
use Proc::Queue size => 3; my $mech = WWW::Mechanize->new(timeout => 90); while (@urllist) { $pid = fork; if(defined ($pid) and $pid==0) { $mech->get($url); print "$mech->content()->as_HTML(); exit(0); } 1 while waitpid(-1, WNOHANG)>0; # reaps childs }
or do I have to
use Proc::Queue size => 3; while (@urllist) { $pid = fork; if(defined ($pid) and $pid==0) { my $mech = WWW::Mechanize->new(timeout => 90); $mech->get($url); print "$mech->content()->as_HTML(); exit(0); } 1 while waitpid(-1, WNOHANG)>0; # reaps childs }
?

Many thanks...

Steve

Replies are listed 'Best First'.
Re: Fork and WWW::Mechanize: Can agent be shared?
by andyford (Curate) on Feb 18, 2007 at 17:16 UTC

    You can do it the first way. But after the fork it's no longer the same object, it's a copy, so you can't share info.

    non-Perl: Andy Ford

Re: Fork and WWW::Mechanize: Can agent be shared?
by shmem (Chancellor) on Feb 18, 2007 at 21:06 UTC
    As said before, the first way should work - but I really wouldn't place the waitpid inside the loop, but move that into a $SIG{CHLD} handler.

    --shmem

    _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                  /\_¯/(q    /
    ----------------------------  \__(m.====·.(_("always off the crowd"))."·
    ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}
Re: Fork and WWW::Mechanize: Can agent be shared?
by perrin (Chancellor) on Feb 18, 2007 at 17:17 UTC
    It can be shared, because it won't actually be shared, if you see what I mean. The only thing you have to worry about with shared objects and forking is forking after opening a socket or file, which I don't think will happen here.
Re: Fork and WWW::Mechanize: Can agent be shared?
by ikegami (Pope) on Feb 19, 2007 at 18:18 UTC
    You'll probably find Parallel::ForkManager useful.
    use Parallel::ForkManager qw( ); use constant MAX_CHILDREN => 3; { my $mech = WWW::Mechanize->new(timeout => 90); my @urllist = ...; my $pm = Parallel::ForkManager->new(MAX_CHILDREN); foreach my $url (@urllist) { # Forks and returns the pid for the child. my $pid = $pm->start() and next; $mech->get($url); print $mech->content()->as_HTML(); # Exit child. $pm->finish(); } }
Re: Fork and WWW::Mechanize: Can agent be shared?
by TOD (Friar) on Feb 19, 2007 at 04:15 UTC
    if you take a shared memory segment you can store a serialized form of the object you want to be common to all processes on it. it must be serialized, because you can't store references on a shmem segment.

    but basically i would prefer threads instead of forking.
Re: Fork and WWW::Mechanize: Can agent be shared?
by jdrago_999 (Hermit) on Feb 19, 2007 at 18:22 UTC
    If you need to share data between multiple processes, take a look at the forks and forks::shared modules (as well as the threads and threads::shared modules).

    The semantics are different than what you're doing now, and there is some extra overhead, but I've been using both threads and forks for a couple years now with no problem, in production environments no less.
Re: Fork and WWW::Mechanize: Can agent be shared?
by tphyahoo (Vicar) on Feb 20, 2007 at 13:20 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://600716]
Front-paged by andyford
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (2)
As of 2021-06-14 03:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    What does the "s" stand for in "perls"? (Whence perls)












    Results (60 votes). Check out past polls.

    Notices?