Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Re: What is this "Do you need to predeclare croak" about?

by marioroy (Prior)
on Jun 14, 2017 at 21:44 UTC ( [id://1192837]=note: print w/replies, xml ) Need Help??


in reply to What is this "Do you need to predeclare croak" about? [SOLVED]

Hello karlgoethebier,

The demonstration looks fine. Some modules may not play well with threads, unfortunately. The LWP::Simple module has many dependencies. One of them may be unsafe for use with threads. In that case, the use_threads option is necessary to have workers spawn via fork on the Windows platform or when loading threads at the top of the script.

Network related tasks may benefit from MCE's interval option. It helps stagger the immediate code that follows. For this use-case, calling yield prevents workers from initiating remote connections at the same time. It is similarly to sleep, but runs serially, not parallel, for that duration of time. All participating workers wait their turn to sleep.

The next MCE update 1.830 will default to 1 for the posix_exit option. It is nearly impossible to manage a list of modules thread-safe or multi-process END safe for that matter.

#!/usr/bin/env perl # http://www.perlmonks.org/?node_id=1192821 use strict; use warnings; use MCE::Loop; use MCE::Shared; use LWP::Simple; use feature qw(say); my $result = MCE::Shared->hash; my @urls = qw(http://perlmonks.org http://www.whitehouse.org); MCE::Loop::init { max_workers => 'auto', chunk_size => 1, interval => 0.008, posix_exit => 1, use_threads => 0 }; my $fetch = sub { eval { head(shift) }; warn $@ if $@; }; mce_loop { MCE->yield; my @data = $fetch->( $_ ); $result->set( $_ => \@data ); } @urls; { no warnings qw(uninitialized); my $iter = $result->iterator(); while ( my ( $url, $data ) = $iter->() ) { say $url; say for @$data; say q(---); } }

MCE Loop is wantarray-aware. This allows one to use the gather method to send the key-value pair into a plain hash. For readers, this is how it was done before MCE::Shared came about.

#!/usr/bin/env perl # http://www.perlmonks.org/?node_id=1192821 use strict; use warnings; use MCE::Loop; use LWP::Simple; use feature qw(say); my @urls = qw(http://perlmonks.org http://www.whitehouse.org); MCE::Loop::init { max_workers => 'auto', chunk_size => 1, interval => 0.008, posix_exit => 1, use_threads => 0 }; my $fetch = sub { eval { head(shift) }; warn $@ if $@; }; my %result = mce_loop { MCE->yield; my @data = $fetch->( $_ ); MCE->gather( $_ => \@data ); } @urls; { no warnings qw(uninitialized); while ( my ( $url, $data ) = each %result ) { say $url; say for @$data; say q(---); } }

Regards, Mario

Replies are listed 'Best First'.
Re^2: What is this "Do you need to predeclare croak" about?
by marioroy (Prior) on Jun 14, 2017 at 23:03 UTC

    Workers yielding serially matters more when involving an event loop. For example preventing 200 workers x 300 chunk size from initiating many connections simultaneously.

    sub walk { my ( $job, $result, $failed ) = @_; # Yielding is critical when running an event loop in parallel. # Not doing so means that the app may reach contention points # with the firewall and likely impose unnecessary hardship at # the OS level. The idea here is not to have multiple workers # initiate HTTP requests to a batch of URLs at the same time. # Yielding in 1.827+ behaves more like scatter for the worker # to run solo in a fraction of time. MCE::Hobo->yield( 0.03 ); # MCE::Hobo 1.827 my $cv = AnyEvent->condvar(); # Populate the hash ref for URLs it could reach. # Do not mix AnyEvent timeout and Hobo timeout. # Choose to do the event timeout if available. foreach my $url ( @{ $job->{INPUT} } ) { $cv->begin(); http_get $url, timeout => 2, sub { my ( $data, $headers ) = @_; $result->{$url} = $data; $cv->end(); }; } $cv->recv(); # Populate the array ref for URLs it could not reach. foreach my $url ( @{ $job->{INPUT} } ) { push @{ $failed }, $url unless (exists $result->{ $url }); } return; }

    Regards, Mario

Re^2: What is this "Do you need to predeclare croak" about?
by karlgoethebier (Abbot) on Jun 15, 2017 at 15:58 UTC

    Thank you very much Mario and best regards, Karl

    «The Crux of the Biscuit is the Apostrophe»

    Furthermore I consider that Donald Trump must be impeached as soon as possible

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1192837]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others exploiting the Monastery: (4)
As of 2024-03-29 00:25 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found