Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

IPC:Shareable: Not an array reference

by Bloehdian (Beadle)
on Oct 10, 2016 at 19:26 UTC ( [id://1173673]=perlquestion: print w/replies, xml ) Need Help??

Bloehdian has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I wanted to share an array between to processes created by fork(), but the following code is broken and I have no clue why:

use strict;
use diagnostics;
use IPC::Shareable;
use POSIX ":sys_wait_h";

my @data;
my %options = (
    create    => 1,
    exclusive => 0,
    mode      => 0644,
    destroy   => 1,
);
my $pid;
my $item;

   if ( ! defined ( $pid = fork() ) ) {
   
      die( "Cannot fork!: $!" )

   }
   elsif ( $pid == 0 ) {
      
      # Child
      #tie( @data, 'IPC::Shareable', 'glue', \%options );
      sleep 1;

      for ( my $i=1; $i <= 10; $i++ ) {
         
         $item = shift( @data );
         if ( ! defined $item ) { $item = '' }
            
         print "Child: $item\n";
         
      }
      
   }
   else {
      
      # Parent
      tie( @data, 'IPC::Shareable', 'glue', \%options );
      
      for ( my $i=1; $i <= 10; $i++ ) {
         
         push( @data, $i );
         print "Parent: @data\n";
         
      }

      waitpid( $pid, WNOHANG );
}

[download]
It yields: 
Not an ARRAY reference at /usr/local/share/perl/5.20.2/IPC/Shareable.p
+m line
        362 (#1)
    (F) Perl was trying to evaluate a reference to an array value, but
+ found
    a reference to something else instead.  You can use the ref() func
+tion
    to find out what kind of ref it really was.  See perlref.

Uncaught exception from user code:
        Not an ARRAY reference at /usr/local/share/perl/5.20.2/IPC/Sha
+reable.pm line 362.
        IPC::Shareable::PUSH(IPC::Shareable=HASH(0x2604088), 1) called
+ at test_2.pl line 45
xyz@v32470:~/trans_gpsd/bin$ Child:
Child:
Child:
Child:
Child:
Child:
Child:
Child:
Child:
Child:

[download]

Any idea?

Cheers

Bloehdian

Replies are listed 'Best First'.
Re: IPC:Shareable: Not an array reference
by marioroy (Prior) on Oct 11, 2016 at 02:47 UTC

    Hi Bloehdian,

    Re: I want to share an array between two processes created by fork()

    The following demonstration provides a cross platform solution. MCE::Shared supports Unix (e.g: AIX, Darwin, FreeBSD, Linux, NetBSD, OpenBSD, Solaris, and possibly others), Windows (ActiveState/Strawberry Perl), and Cygwin. Also supported is Perl included with git-shell for Windows.

    use strict;
use warnings;

use MCE::Shared;

tie my @data, 'MCE::Shared';

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   my $item;
   sleep 1;

   for ( my $i = 1; $i <= 10; $i++ ) {
      $item = shift @data;
      $item = '' if ( ! defined $item );

      print "Child: $item\n";
   }
}
else {
   # Parent
   for ( my $i = 1; $i <= 10; $i++ ) {
      push @data, $i;
      print "Parent: @data\n";
   }

   waitpid( $pid, 0 );
}

    [download]

    The snippet produces the following output.

    Parent: 1
Parent: 1 2
Parent: 1 2 3
Parent: 1 2 3 4
Parent: 1 2 3 4 5
Parent: 1 2 3 4 5 6
Parent: 1 2 3 4 5 6 7
Parent: 1 2 3 4 5 6 7 8
Parent: 1 2 3 4 5 6 7 8 9
Parent: 1 2 3 4 5 6 7 8 9 10
Child: 1
Child: 2
Child: 3
Child: 4
Child: 5
Child: 6
Child: 7
Child: 8
Child: 9
Child: 10

    [download]

    Regards, Mario.

[reply]
[d/l]
[select]

      Calling MCE::Shared->init() is recommended whenever running many workers simultaneously. This is a one-time call which makes a request to the shared-manager process for a data channel to use for IPC. Init is called automatically by MCE workers so not necessary at the application level.

      Init is helpful for extra performance and applies to running with many workers only.

      elsif ( $pid == 0 ) {

   # Child
   MCE::Shared->init();

   ...
}

      [download]

      Regards, Mario.

[reply]
[d/l]

        Hello Mario,

        THX for the reply.

        I checked the documentation on CPAN, but it is not totally clear to me, how I would implement locking with this module. In the actual application, both processes will manipulate @data.

        Cheers

        Bloehdian

[reply]
[reply]

        Hello Marshall,

        Supporting the Windows platform took many attempts and am embarrassed to say how many hours spent just so that MCE::Shared works there too. What is cool about MCE::Shared is the ability to share data no matter how workers are spawned (e.g. via threads, via fork, MCE, MCE::Hobo, Parallel::ForkManager, and possibly other parallel modules).

        The MCE module supports Perl 5.8 and above whereas MCE::Shared requires Perl 5.10.1. This is the reason the two modules are separated. There are still users out there wanting MCE to run on Perl 5.8 minimally (e.g. App::Netdisco, Test::Perl::Critic).

        Regards, Mario.

[reply]
Re: IPC:Shareable: Not an array reference
by marioroy (Prior) on Oct 11, 2016 at 09:15 UTC

    After writing a solution using MCE::Shared, I wanted to come back and give IPC::Shareable a try including locking. Here it is.

    use strict;
use warnings;

use IPC::Shareable;

my ( $glue, $options ) = ( 'data', {
   create    => 1,
   exclusive => 0,
   mode      => 0644,
   destroy   => 1,
});

tie my @data, 'IPC::Shareable', $glue, $options;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   my $item;
   sleep 1;

   for ( my $i = 1; $i <= 10; $i++ ) {
      ( tied @data )->shlock;
      $item = shift @data;
      ( tied @data )->shunlock;

      $item = '' if ( ! defined $item );
      print "Child: $item\n";
   }
}
else {
   # Parent
   for ( my $i = 1; $i <= 10; $i++ ) {
      ( tied @data )->shlock;
      push @data, $i;
      ( tied @data )->shunlock;

      print "Parent: @data\n";
   }

   waitpid( $pid, 0 );
}

    [download]

    It produces the following output on FreeBSD, Linux, and Mac OS X.

    Parent: 1
Parent: 1 2
Parent: 1 2 3
Parent: 1 2 3 4
Parent: 1 2 3 4 5
Parent: 1 2 3 4 5 6
Parent: 1 2 3 4 5 6 7
Parent: 1 2 3 4 5 6 7 8
Parent: 1 2 3 4 5 6 7 8 9
Parent: 1 2 3 4 5 6 7 8 9 10
Child: 1
Child: 2
Child: 3
Child: 4
Child: 5
Child: 6
Child: 7
Child: 8
Child: 9
Child: 10

    [download]

    Notice how the construction for the shared variable is done before spawning. Also, the parent waits for the spawned worker to exit. Other than locking (optional for this use-case due to sleep in worker), everything else is similar to the OP's code.

    Regards, Mario.

[reply]
[d/l]
[select]

      The following is a comparison demonstrating 2 workers incrementing a shared counter variable. The OO interface for MCE::Shared allows for zero-locking at the application level. All 3 snippets involve set and get while incrementing the variable.

      1) ipc_shareable.pl

      use strict;
use warnings;

use IPC::Shareable;

my $options = {
   create    => 1,
   exclusive => 0,
   mode      => 0644,
   destroy   => 1,
};

tie my $cntr, 'IPC::Shareable', $options;
my $val;

$cntr = 0;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   for ( 1 .. 10000 ) {
      ( tied $cntr )->shlock;
      $val = $cntr++;
      ( tied $cntr )->shunlock;
   }
}
else {
   # Parent
   for ( 1 .. 10000 ) {
      ( tied $cntr )->shlock;
      $val = $cntr++;
      ( tied $cntr )->shunlock;
   }

   waitpid $pid, 0;

   print "counter: ", $cntr, "\n";
}

      [download]

      2) mce_shared.pl

      use strict;
use warnings;

use MCE::Mutex;
use MCE::Shared;

my $mutex = MCE::Mutex->new();
tie my $cntr, 'MCE::Shared', 0;
my $val;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   for ( 1 .. 10000 ) {
      $mutex->lock;
      $val = $cntr++;
      $mutex->unlock;
   }
}
else {
   # Parent
   for ( 1 .. 10000 ) {
      $mutex->lock;
      $val = $cntr++;
      $mutex->unlock;
   }

   waitpid( $pid, 0 );

   print "counter: ", $cntr, "\n";
}

      [download]

      3) mce_shared_oo.pl

      use strict;
use warnings;

use MCE::Shared;

my $cntr = MCE::Shared->scalar( 0 );
my $val;

if ( ! defined ( my $pid = fork() ) ) {
   die "Cannot fork!: $!";
}
elsif ( $pid == 0 ) {
   # Child
   for ( 1 .. 10000 ) {
      $val = $cntr->incr;
   }
}
else {
   # Parent
   for ( 1 .. 10000 ) {
      $val = $cntr->incr;
   }

   waitpid( $pid, 0 );

   print "counter: ", $cntr->get, "\n";
}

      [download]

      Results

      time in seconds  :  FreeBSD  Linux  Mac OSX

ipc_shareable.pl :   1.80    1.87    1.79
mce_shared.pl    :   0.68    0.97    0.58
mce_shared_oo.pl :   0.38    0.73    0.41

counter: 20000

      [download]

      Fetches require more time on Linux than BSD-based OS in regards to sockets. This is seen not only with Perl/MCE::Shared but also with other languages. I'm not sure why.

      MCE::Shared is fully wantarray-aware and performs fetches only when requested. Sadly, that is not the case via the TIE interface. Perhaps, this is an unnecessary overhead IMHO in regards to the TIE implementation inside Perl.

      The following results were obtained by omitting fetches: e.g. $cntr++, $cntr->incr. Fortunately, stores perform similarly between BSD and Linux via the OO interface.

      time in seconds  :  FreeBSD  Linux  Mac OSX

ipc_shareable.pl :   1.77    1.86    1.61
mce_shared.pl    :   0.68    0.95    0.58
mce_shared_oo.pl :   0.19    0.17    0.16

counter: 20000

      [download]

      The above is only a glimpse of what is possible with MCE::Shared. The use of parallel data-channels (enabled automatically) for MCE, MCE::Hobo, and threads further reduce latency for the shared-manager process. Calling MCE::Shared->init(), inside the worker, enables the optimization for fork and other parallel modules.

      This has been an interesting exercise. Thank you Bloehdian for the initial post.

      Tests were conducted on a Macbook Pro laptop (Haswell architecture) OS X 10.11.6 running at 2.6 GHz 1600 MHz RAM. FreeBSD and Linux are virtual machines managed by Parallels Desktop 11.2.1.

      Regards, Mario.

[reply]
[d/l]
[select]
Re: IPC:Shareable: Not an array reference
by Marshall (Canon) on Oct 11, 2016 at 00:28 UTC
    I don't have a Unix machine right now to test with, only Windows. I think Windows does some kind of fork emulation rather than a real separate process fork so I'm not sure that any test I run would be completely valid.

    Anyway, from looking at your code, the first thing that jumps out at me is: where are the locks? If you have shared memory, your processes must implement some form of locking otherwise you have a mess where 2 folks are trying to modify the same structure at once. It is up to the user to implement these locks between processes. IPC:Shareable shows the shlock() and shunlock() functions.  shift @data; is a "read-modify-write" operation.

    Update: Oh, I see I guess you are figuring that the sleep(1) in the child is enough?
    From looking at the example in the doc's, I think you need different values for "create" and "destroy" between the 2 processes. Why is tie commented out in the child? Shouldn't the sleep 1 go before the tie to allow time for the parent to create this shared array in the first place? I would simplify things and replace the for loop in the child with a simple, print "@data";.

[reply]
[d/l]
[select]

Back to Seekers of Perl Wisdom

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?

www.com | www.net | www.org
Node Status?
node history
Node Type: perlquestion [id://1173673]
Approved by Discipulus
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? • Last hour • Other CB clients
Other Users?
Others examining the Monastery: (5)
As of 2024-04-23 15:28 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found