Where does the 6th process come from? [SOLVED]

[karlgoethebier]

Good day all,

this is just another useless waste of time for fun and perhaps learning.

I spawn 4 processes but ps shows 6. But it should be 5, right?

#!/usr/bin/env perl

# "When I first started hobo'in,
# I took a freight train to be my friend"
# -John Lee Hooker
# https://www.youtube.com/watch?v=1kQlRQRGdfQ

use strict;
use warnings;
use MCE::Hobo;
use MCE::Shared;
use feature qw(say);
# use Sereal ();
use MCE::Mutex;

say "procs at start:\n", procs();

our $mutex = MCE::Mutex->new;

# say $mutex->impl();
my $result = MCE::Shared->array();
my $foo    = MCE::Hobo->create( \&task, "foo" );
my $bar    = MCE::Hobo->create( \&task, "bar" );
my $nose   = MCE::Hobo->create( \&task, "nose" );
my $cuke   = MCE::Hobo->create( \&task, "cuke" );
my @hobos  = MCE::Hobo->list();

say "spawned hobos:";

for (@hobos) {
    if ( $_->is_running() ) {
        say 'pid: ', $_->pid();
    }
}

say 'pending hobos: ', MCE::Hobo->pending();
say "procs at pending:\n", procs();

$_->join() for @hobos;

say "procs after join:\n", procs();
say "result:";
say for @$result;

sub task {
    our $mutex;
    $mutex->lock;
    $result->push(shift);
    $mutex->unlock;
}

sub procs {
    my $procs = qx(ps aux | grep [p]erl);
    $procs =~ s/ //g;
    chomp $procs;
    $procs;
}
__END__

karls-mac-mini:MyAdaModule karl$ ./hobo.pl
procs at start:
karl254417,30,124421128816s000S+6:03pm0:00.07perl./hobo.pl
spawned hobos:
pid: 2549
pid: 2550
pid: 2551
pid: 2552
pending hobos: 4
procs at pending:
karl254417,30,124421129264s000S+6:03pm0:00.08perl./hobo.pl
karl25520,00,000s000Z+6:03pm0:00.00(perl)
karl25510,00,000s000Z+6:03pm0:00.00(perl)
karl25500,00,000s000Z+6:03pm0:00.00(perl)
karl25490,00,000s000Z+6:03pm0:00.00(perl)
karl25480,00,024421123116s000S+6:03pm0:00.00perl./hobo.pl
procs after join:
karl254417,30,124421129264s000S+6:03pm0:00.08perl./hobo.pl
karl25480,00,024421123136s000S+6:03pm0:00.00perl./hobo.pl
result:
foo
bar
nose
cuke

Update: Added more debugging output

Thanks for any hint and sorry if i missed something essential.

Update 2:

#!/usr/bin/env perl

use strict;
use warnings;

use threads;
use threads::shared;
use MCE::Hobo;
use MCE::Shared;
use feature qw(say);

say q(procs at start: ), procs();

my $result = MCE::Shared->array();

for (qw(foo bar nose cuke)) {
    MCE::Hobo->create( \&task, $_ );
}

my @hobos = MCE::Hobo->list();

say q(spawned hobos:);

for (@hobos) {
    if ( $_->is_running() ) {
        say q(pid: ), $_->pid();
    }
}

say q(pending hobos: ), MCE::Hobo->pending();

say q(procs at pending: ), procs();

$_->join() for @hobos;

say q(procs after join: ), procs();

say q(result:);

say for @$result;

sub task {
    $result->push(shift);
}

sub procs {
    my $procs = grep { /perl/ } qx(ps aux);
}

__END__

karls-mac-mini:monks karl$  ./hobo.pl
procs at start: 1
spawned hobos:
pid: 3835
pid: 3836
pid: 3837
pid: 3838
pending hobos: 4
procs at pending: 5
procs after join: 1
result:
foo
bar
nose
cuke

Regards, Karl

[marioroy]

On Unix platforms including Cygwin, the MCE::Shared module spawns a shared-manager process unless threads is present.

1 for the main process
1 for the shared-manager process (a thread when threads is present)
4 for the hobo processes

The Sereal modules (Decoder/Encoder) are loaded automatically. Basically, the Storable module is not loaded when Sereal 3.015 or later is available. The following are recommended modules. IO::FDPass allows the passing of the fd descriptor while the shared-manager is running.

## MCE and MCE::Shared including MCE::Hobo

1. Sereal::Decoder 3.015+
2. Sereal::Encoder 3.015+
3. Sereal (ok for completeness, but MCE does not load this)

## MCE::Shared applies to Condvar, Handle, and Queue

1. IO::FDPass 1.2+

[karlgoethebier]

"...unless threads is present"

Thank you very much marioroy!

But threads is present:

karls-mac-mini:monks karl$ cpanm threads
threads is up to date. (2.15)

Best regards, Karl

[marioroy]

... if present meaning if threads is loaded inside the script.

use threads;

...

On Unix platforms including Cygwin, one can use threads and threads::shared alongside MCE::Shared. In that case, MCE::Shared spawns a thread for the shared-manager. For the Windows platform, the MCE modules loads threads automatically for extra stability.

Below, more clarity for the recommended modules.

(Q) Why is Sereal loaded transparently?

(A) The main reason is for extra performance. To ensure minimum memory consumption, there's no reason to load the Storable module if Sereal is available in Perl. This is handled transparently.

(Q) Why is IO::FDPass beneficial?

(A) Being able to construct a shared condvar, handle, or queue while the shared-manager is running greatly adds to the ease-of-use. These involve handles behind the scene. Basically, am able to send the relevant fd descriptors to the shared-manager. Without FDPass, one must be careful to construct these shared objects first and later start the shared-manager manually.

[marioroy]

For shared objects, the OO interface has the least overhead. MCE::Shared provides on-demand dereferencing for array and hash-like behavior. In that case, a mutex is needed whenever dereferencing involves a fetch and store (equals 2 IPC trips to the shared-manager). Regarding the OO interface, there is a single entry point to the shared-manager process. This is why a mutex isn't needed when accessing shared objects via the OO interface, unless wanting a mutex around updating two or more shared objects.

sub task {
    our $mutex;
    $mutex->lock; # not necessary when accessing shared-objects via th
+e OO interface
    $result->push(shift);
    $mutex->unlock;

    $mutex->lock; # necessary if dereferencing involves fetch and stor
+e behind TIE
    push @{ $result }, shift;
    $mutex->unlock;
}

[marioroy]

Please find below various demonstrations. Increase the loop iteration from 1e5 to 1e6 and watch top or task manager.

#!/usr/bin/env perl

use strict;
use warnings;
use MCE::Hobo;
use MCE::Shared;
use Time::HiRes qw(time);
use feature qw(say);

my $number = MCE::Shared->scalar(0);
my $start  = time;

MCE::Hobo->create('task', $_) for 1 .. 4;
MCE::Hobo->waitall;

printf "duration: %0.3f seconds\n", time - $start;
say "number: ", $number->get();

sub task {
    my ($id) = @_;
    say "Hobo $id started";
    $number->incr() for 1 .. 100000;
    say "Hobo $id ended";
}

__END__

Hobo 1 started
Hobo 2 started
Hobo 3 started
Hobo 4 started
Hobo 4 ended
Hobo 2 ended
Hobo 1 ended
Hobo 3 ended
duration: 0.872 seconds
number: 400000

Fetching data involves extra latency due to workers waiting for a response from the shared-manager process. No worries, it runs reasonably well for being a plain-Perl module.

sub task {
    my ($id, $val) = @_;
    say "Hobo $id started";
    $val = $number->incr() for 1 .. 100000;
    say "Hobo $id ended";
}

__END__

Hobo 1 started
Hobo 2 started
Hobo 3 started
Hobo 4 started
Hobo 4 ended
Hobo 3 ended
Hobo 1 ended
Hobo 2 ended
duration: 2.183 seconds
number: 400000

Dereferencing involving a fetch and store requires a mutex. Overhead goes up the roof. There's now a mutex lock, TIE STORE + FETCH (2 IPCs for that), and finally a mutex unlock.

#!/usr/bin/env perl

use strict;
use warnings;
use MCE::Hobo;
use MCE::Mutex;
use MCE::Shared;
use Time::HiRes qw(time);
use feature qw(say);

my $mutex  = MCE::Mutex->new( impl => 'Channel' );
my $number = MCE::Shared->scalar(0);
my $start  = time;

MCE::Hobo->create('task', $_) for 1 .. 4;
MCE::Hobo->waitall;

printf "duration: %0.3f seconds\n", time - $start;
say "number: ", $number->get();

sub task {
    my ($id, $val) = @_;
    say "Hobo $id started";
    for ( 1 .. 100000 ) {
       $mutex->lock;
       $val = ++${ $number };
       $mutex->unlock;
    }
    say "Hobo $id ended";
}

__END__

Hobo 1 started
Hobo 2 started
Hobo 3 started
Hobo 4 started
Hobo 2 ended
Hobo 1 ended
Hobo 4 ended
Hobo 3 ended
duration: 20.343 seconds
number: 400000

Let's remove dereferencing. Thus, mutex and OO only.

sub task {
    my ($id, $val) = @_;
    say "Hobo $id started";
    for ( 1 .. 100000 ) {
       $mutex->lock;
       $val = $number->incr();
       $mutex->unlock;
    }
    say "Hobo $id ended";
}

__END__

Hobo 1 started
Hobo 2 started
Hobo 3 started
Hobo 4 started
Hobo 3 ended
Hobo 4 ended
Hobo 1 ended
Hobo 2 ended
duration: 9.702 seconds
number: 400000

Summary

Performance is possible via the OO interface. Dereferencing is nice if you need it. Mutex is possible as well for synchronization. Any fetch from the shared-manager takes extra time. MCE::Shared is fully wantarray aware. Meaning, the MCE::Shared engine will only do extra stuff whenever needed, not more.

Regards, Mario.

[karlgoethebier]

Thank you marioroy!

[shmem]

The grep.

- my $procs = qx(ps aux | grep [p]erl | wc -l);
+ my $procs = qx(ps aux | grep [p]erl | grep -v grep | wc -l);

update: Scratch that. The [p] avoids the grep to be included.

[Anonymous Monk]

The [p] should obviate the need for grep -v, but maybe you should use quotes around the pattern: grep '[p]erl', or better yet,

my $procs = grep /perl/, qx(ps aux);

[Anonymous Monk]

It shows 5 on my machine, not 6. Why don't you check the procs() at the top of your program, before you start hoboing?

It would be a whole lot easier to read the output if you would

say 'procs ', procs();

so we wouldn't have to guess which output line goes with which say.