Re^3: Child process inter communication

Please know that the results on this post doesn't count. I've overclocked a system on purpose, mainly for simulating a fast machine in the future. On an 8-core box (total 16 logical cores), I ran the same tests to better understand the impact between the three implementations. The box is overclocked to 4.0 GHz. What is interesting to me is not how fast but rather the impact from running many workers.

Baseline testing involves 4 workers and runs about 2x faster than my laptop. No surprises there due to running at 4 GHz and not from a Linux VM.

Update: Added results for MCE::Inbox which will ship with MCE::Shared 1.827 on release day. It is Inbox2 with optimizations.

# 4 GHz, CentOS 7 ( Perl v5.16.3 )

$ perl foo1.pl | wc -l      # Foo::Inbox
duration:  0.441 seconds
   50000

$ perl inbox.pl | wc -l     # MCE::Inbox w/ blocking capability
duration:  0.422 seconds
   50000

$ perl foo2.pl | wc -l      # MCE::Shared->queue
duration:  0.645 seconds
   50000

$ perl foo4.pl | wc -l      # Thread::Queue
duration:  1.544 seconds
   50000
[download]

Next is 128 workers retrieving 1.6 million messages.

# 4 GHz, CentOS 7 ( Perl v5.16.3 )

$ perl foo1.pl | wc -l      # Foo::Inbox
duration: 12.685 seconds
   1600000

$ perl inbox.pl | wc -l     # MCE::Inbox w/ blocking capability
duration: 15.939 seconds
   1600000

$ perl foo2.pl | wc -l      # MCE::Shared->queue
duration: 21.533 seconds
   1600000

$ perl foo4.pl | wc -l      # Thread::Queue
duration: 90.015 seconds
   1600000
[download]

Unfortunately, threads may not scale linearly, especially when involving locking. In top, I see the process peak at 306% max while running. It's far from the 1600% mark hardware limit. On the other hand, MCE::Inbox scales nicely considering it does blocking as well.

For the 1.6 million test, I added a loop to generate more names, 128 total.

my @names = shuffle qw/ Barny Betty Fred Wilma /;

foreach my $i ( 1 .. 5 ) {
   push @names, map { $_.$i } @names;
   print scalar(@names), "\n";
}

__END__

8
16
32
64
128
[download]

There was one other change and replaced 4167 with 98 because there are now 128 workers, not 4.

      ...
      # send greeting again
      $inbox->send($name, \@names, 'Hello') if $count < 98;

      # eventually stop benchmarking
      last if ++$count == 12500;
      ...
[download]

Regards, Mario

Comment on Re^3: Child process inter communication Select or Download Code

Replies are listed 'Best First'.

Re^4: Child process inter communication
by marioroy (Prior) on Jan 23, 2018 at 09:34 UTC

MCE::Inbox lives on Github, currently. Depending on whether threads is loaded, it will configure channels using Thread::Queue or MCE::Shared::Queue otherwise. There are examples demonstrating "Chameneos, a Concurrency Game for Java, Ada, and Others" using Perl over here.

I will make a MCE::Inbox release on CPAN after MCE 1.834 and MCE::Shared 1.835. These provide a fix on Microsoft Windows so that dequeue doesn't involve checking the socket each time if ready. On Windows, I figured a way so that nested-sessions may continue to work and not degrade dequeue performance from Perl 5.20 and up. Basically, workers spawning workers or workers exiting while other workers dequeue items from a queue.

Cheers, Mario

[reply]


XP is just a number
	PerlMonks