Beefy Boxes and Bandwidth Generously Provided by pair Networks
There's more than one way to do things
 
PerlMonks  

Re^3: OT: Parallel::ForkManager and suitable load (Ubuntu)

by marioroy (Prior)
on Mar 12, 2024 at 16:48 UTC ( [id://11158206]=note: print w/replies, xml ) Need Help??


in reply to Re^2: OT: Parallel::ForkManager and suitable load (Ubuntu)
in thread OT: Parallel::ForkManager and suitable load (Ubuntu)

> Interestingly, the original server had stability problems, ...

Greetings, talexb. That can be from many kids initiating a connection simultaneously in a tiny window. The following is a P::FM like code. The interesting bit is MCE::Hobo->yield, helpful for staggering connection instantiations. MCE::Hobo->yield defaults to 0.008 seconds on UNIX. Call yield prior to making a remote connection.

use v5.10.1; use MCE::Hobo; MCE::Hobo->init( max_workers => 50, posix_exit => 1, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Hobo->create( $data, sub { MCE::Hobo->yield(0.008); # sleep 1; # simulate connection instantiation [ $data * 2 ]; }); } MCE::Hobo->wait_all;

Without the "sleep 1", the demonstration completes in ~ 0.8 seconds (without yield) and ~ 16.0 seconds (with yield). Otherwise, ~ 42 seconds simulating work. No matter how many workers, the serial-delay capability will not allow more than 125 connection instantiations per second with MCE::Hobo->yield(); default 0.008 seconds.

For use-cases like this, serialized delay/yield mitigates many workers initiating a connection concurrently in a tiny window, improving reliability.

Replies are listed 'Best First'.
Re^4: OT: Parallel::ForkManager and suitable load (Ubuntu)
by marioroy (Prior) on Mar 12, 2024 at 17:17 UTC

    MCE::Hobo uses MCE::Shared for IPC. You may prefer MCE::Child to not involve a shared-manager process. MCE::Child uses MCE::Channel for IPC. MCE::Channel came long after making MCE::Hobo and thought to make something similar that works with Perl v5.8.1 and above.

    use v5.10.1; use MCE::Child; MCE::Child->init( max_workers => 50, posix_exit => 1, on_finish => sub { my ($pid, $exit_code, $ident, $exit_signal, $error, $resp) = @ +_; print "child $pid completed: $ident => ", $resp->[0], "\n"; } ); foreach my $data ( 1..2000 ) { MCE::Child->create( $data, sub { MCE::Child->yield(0.008); # sleep 1; # simulate connection instantiation [ $data * 2 ]; }); } MCE::Child->wait_all;

    Writing MCE::Child was helpful in improving MCE::Hobo and vice versa. Yeah, there are two similar modules. MCE::Child is mostly compatible with MCE::Hobo. MCE::Hobo is more compatible with threads, but requires Perl v5.10.1 minimally.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://11158206]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others surveying the Monastery: (1)
As of 2025-02-09 14:15 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Which URL do you most often use to access this site?












    Results (96 votes). Check out past polls.