Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Where does the 6th process come from? [SOLVED]

by karlgoethebier (Prior)
on Apr 22, 2017 at 14:24 UTC ( #1188619=perlquestion: print w/replies, xml ) Need Help??
karlgoethebier has asked for the wisdom of the Perl Monks concerning the following question:

Good day all,

this is just another useless waste of time for fun and perhaps learning.

I spawn 4 processes but ps shows 6. But it should be 5, right?

#!/usr/bin/env perl # "When I first started hobo'in, # I took a freight train to be my friend" # -John Lee Hooker # https://www.youtube.com/watch?v=1kQlRQRGdfQ use strict; use warnings; use MCE::Hobo; use MCE::Shared; use feature qw(say); # use Sereal (); use MCE::Mutex; say "procs at start:\n", procs(); our $mutex = MCE::Mutex->new; # say $mutex->impl(); my $result = MCE::Shared->array(); my $foo = MCE::Hobo->create( \&task, "foo" ); my $bar = MCE::Hobo->create( \&task, "bar" ); my $nose = MCE::Hobo->create( \&task, "nose" ); my $cuke = MCE::Hobo->create( \&task, "cuke" ); my @hobos = MCE::Hobo->list(); say "spawned hobos:"; for (@hobos) { if ( $_->is_running() ) { say 'pid: ', $_->pid(); } } say 'pending hobos: ', MCE::Hobo->pending(); say "procs at pending:\n", procs(); $_->join() for @hobos; say "procs after join:\n", procs(); say "result:"; say for @$result; sub task { our $mutex; $mutex->lock; $result->push(shift); $mutex->unlock; } sub procs { my $procs = qx(ps aux | grep [p]erl); $procs =~ s/ //g; chomp $procs; $procs; } __END__ karls-mac-mini:MyAdaModule karl$ ./hobo.pl procs at start: karl254417,30,124421128816s000S+6:03pm0:00.07perl./hobo.pl spawned hobos: pid: 2549 pid: 2550 pid: 2551 pid: 2552 pending hobos: 4 procs at pending: karl254417,30,124421129264s000S+6:03pm0:00.08perl./hobo.pl karl25520,00,000s000Z+6:03pm0:00.00(perl) karl25510,00,000s000Z+6:03pm0:00.00(perl) karl25500,00,000s000Z+6:03pm0:00.00(perl) karl25490,00,000s000Z+6:03pm0:00.00(perl) karl25480,00,024421123116s000S+6:03pm0:00.00perl./hobo.pl procs after join: karl254417,30,124421129264s000S+6:03pm0:00.08perl./hobo.pl karl25480,00,024421123136s000S+6:03pm0:00.00perl./hobo.pl result: foo bar nose cuke

Update: Added more debugging output

Thanks for any hint and sorry if i missed something essential.

Update 2:

#!/usr/bin/env perl use strict; use warnings; use threads; use threads::shared; use MCE::Hobo; use MCE::Shared; use feature qw(say); say q(procs at start: ), procs(); my $result = MCE::Shared->array(); for (qw(foo bar nose cuke)) { MCE::Hobo->create( \&task, $_ ); } my @hobos = MCE::Hobo->list(); say q(spawned hobos:); for (@hobos) { if ( $_->is_running() ) { say q(pid: ), $_->pid(); } } say q(pending hobos: ), MCE::Hobo->pending(); say q(procs at pending: ), procs(); $_->join() for @hobos; say q(procs after join: ), procs(); say q(result:); say for @$result; sub task { $result->push(shift); } sub procs { my $procs = grep { /perl/ } qx(ps aux); } __END__ karls-mac-mini:monks karl$ ./hobo.pl procs at start: 1 spawned hobos: pid: 3835 pid: 3836 pid: 3837 pid: 3838 pending hobos: 4 procs at pending: 5 procs after join: 1 result: foo bar nose cuke

Regards, Karl

«The Crux of the Biscuit is the Apostrophe»

Furthermore I consider that Donald Trump must be impeached as soon as possible

Replies are listed 'Best First'.
Re: Where does the 6th process come from?
by marioroy (Curate) on Apr 22, 2017 at 16:12 UTC

    On Unix platforms including Cygwin, the MCE::Shared module spawns a shared-manager process unless threads is present.

    1 for the main process 1 for the shared-manager process (a thread when threads is present) 4 for the hobo processes

    The Sereal modules (Decoder/Encoder) are loaded automatically. Basically, the Storable module is not loaded when Sereal 3.015 or later is available. The following are recommended modules. IO::FDPass allows the passing of the fd descriptor while the shared-manager is running.

    ## MCE and MCE::Shared including MCE::Hobo 1. Sereal::Decoder 3.015+ 2. Sereal::Encoder 3.015+ 3. Sereal (ok for completeness, but MCE does not load this) ## MCE::Shared applies to Condvar, Handle, and Queue 1. IO::FDPass 1.2+
      "...unless threads is present"

      Thank you very much marioroy!

      But threads is present:

      karls-mac-mini:monks karl$ cpanm threads threads is up to date. (2.15)

      Best regards, Karl

      «The Crux of the Biscuit is the Apostrophe»

      Furthermore I consider that Donald Trump must be impeached as soon as possible

        ... if present meaning if threads is loaded inside the script.

        use threads; ...

        On Unix platforms including Cygwin, one can use threads and threads::shared alongside MCE::Shared. In that case, MCE::Shared spawns a thread for the shared-manager. For the Windows platform, the MCE modules loads threads automatically for extra stability.

        Below, more clarity for the recommended modules.

        (Q) Why is Sereal loaded transparently?

        (A) The main reason is for extra performance. To ensure minimum memory consumption, there's no reason to load the Storable module if Sereal is available in Perl. This is handled transparently.

        (Q) Why is IO::FDPass beneficial?

        (A) Being able to construct a shared condvar, handle, or queue while the shared-manager is running greatly adds to the ease-of-use. These involve handles behind the scene. Basically, am able to send the relevant fd descriptors to the shared-manager. Without FDPass, one must be careful to construct these shared objects first and later start the shared-manager manually.


Re: Where does the 6th process come from?
by marioroy (Curate) on Apr 22, 2017 at 16:38 UTC

    For shared objects, the OO interface has the least overhead. MCE::Shared provides on-demand dereferencing for array and hash-like behavior. In that case, a mutex is needed whenever dereferencing involves a fetch and store (equals 2 IPC trips to the shared-manager). Regarding the OO interface, there is a single entry point to the shared-manager process. This is why a mutex isn't needed when accessing shared objects via the OO interface, unless wanting a mutex around updating two or more shared objects.

    sub task { our $mutex; $mutex->lock; # not necessary when accessing shared-objects via th +e OO interface $result->push(shift); $mutex->unlock; $mutex->lock; # necessary if dereferencing involves fetch and stor +e behind TIE push @{ $result }, shift; $mutex->unlock; }

      Please find below various demonstrations. Increase the loop iteration from 1e5 to 1e6 and watch top or task manager.

      #!/usr/bin/env perl use strict; use warnings; use MCE::Hobo; use MCE::Shared; use Time::HiRes qw(time); use feature qw(say); my $number = MCE::Shared->scalar(0); my $start = time; MCE::Hobo->create('task', $_) for 1 .. 4; MCE::Hobo->waitall; printf "duration: %0.3f seconds\n", time - $start; say "number: ", $number->get(); sub task { my ($id) = @_; say "Hobo $id started"; $number->incr() for 1 .. 100000; say "Hobo $id ended"; } __END__ Hobo 1 started Hobo 2 started Hobo 3 started Hobo 4 started Hobo 4 ended Hobo 2 ended Hobo 1 ended Hobo 3 ended duration: 0.872 seconds number: 400000

      Fetching data involves extra latency due to workers waiting for a response from the shared-manager process. No worries, it runs reasonably well for being a plain-Perl module.

      sub task { my ($id, $val) = @_; say "Hobo $id started"; $val = $number->incr() for 1 .. 100000; say "Hobo $id ended"; } __END__ Hobo 1 started Hobo 2 started Hobo 3 started Hobo 4 started Hobo 4 ended Hobo 3 ended Hobo 1 ended Hobo 2 ended duration: 2.183 seconds number: 400000

      Dereferencing involving a fetch and store requires a mutex. Overhead goes up the roof. There's now a mutex lock, TIE STORE + FETCH (2 IPCs for that), and finally a mutex unlock.

      #!/usr/bin/env perl use strict; use warnings; use MCE::Hobo; use MCE::Mutex; use MCE::Shared; use Time::HiRes qw(time); use feature qw(say); my $mutex = MCE::Mutex->new( impl => 'Channel' ); my $number = MCE::Shared->scalar(0); my $start = time; MCE::Hobo->create('task', $_) for 1 .. 4; MCE::Hobo->waitall; printf "duration: %0.3f seconds\n", time - $start; say "number: ", $number->get(); sub task { my ($id, $val) = @_; say "Hobo $id started"; for ( 1 .. 100000 ) { $mutex->lock; $val = ++${ $number }; $mutex->unlock; } say "Hobo $id ended"; } __END__ Hobo 1 started Hobo 2 started Hobo 3 started Hobo 4 started Hobo 2 ended Hobo 1 ended Hobo 4 ended Hobo 3 ended duration: 20.343 seconds number: 400000

      Let's remove dereferencing. Thus, mutex and OO only.

      sub task { my ($id, $val) = @_; say "Hobo $id started"; for ( 1 .. 100000 ) { $mutex->lock; $val = $number->incr(); $mutex->unlock; } say "Hobo $id ended"; } __END__ Hobo 1 started Hobo 2 started Hobo 3 started Hobo 4 started Hobo 3 ended Hobo 4 ended Hobo 1 ended Hobo 2 ended duration: 9.702 seconds number: 400000

      Summary

      Performance is possible via the OO interface. Dereferencing is nice if you need it. Mutex is possible as well for synchronization. Any fetch from the shared-manager takes extra time. MCE::Shared is fully wantarray aware. Meaning, the MCE::Shared engine will only do extra stuff whenever needed, not more.

      Regards, Mario.

      Thank you marioroy!

      «The Crux of the Biscuit is the Apostrophe»

      Furthermore I consider that Donald Trump must be impeached as soon as possible

Re: Where does the 6th process come from?
by Anonymous Monk on Apr 22, 2017 at 15:06 UTC

    It shows 5 on my machine, not 6. Why don't you check the procs() at the top of your program, before you start hoboing?

    It would be a whole lot easier to read the output if you would

    say 'procs ', procs();

    so we wouldn't have to guess which output line goes with which say.

Re: Where does the 6th process come from?
by shmem (Chancellor) on Apr 22, 2017 at 15:18 UTC

    The grep.

    - my $procs = qx(ps aux | grep [p]erl | wc -l); + my $procs = qx(ps aux | grep [p]erl | grep -v grep | wc -l);

    update: Scratch that. The [p] avoids the grep to be included.

    perl -le'print map{pack c,($-++?1:13)+ord}split//,ESEL'
      The [p] should obviate the need for grep -v, but maybe you should use quotes around the pattern: grep '[p]erl', or better yet,
      my $procs = grep /perl/, qx(ps aux);

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1188619]
Approved by marto
Front-paged by stevieb
help
Chatterbox?
[oakbox]: thanks, marto.
[marto]: there was a period where search.cpan had some frequent outages for (IIRC) a couple of weeks. I've not had any problems since, until today.
[marto]: this was about 16 months ago maybe. the noc team are, in my experience, very responsive to reports of issues, so please raise the issue after checking known problems/outages

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2017-07-26 10:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    I came, I saw, I ...
























    Results (390 votes). Check out past polls.