Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW

Re^4: A new CB reader

by marioroy (Vicar)
on Jul 11, 2019 at 01:26 UTC ( #11102656=note: print w/replies, xml ) Need Help??

in reply to Re^3: A new CB reader
in thread A new CB reader

Update: Made a fork so not having to apply the diff by hand if wanting to try The only benefit to using MCE::Child and MCE::Channel is that there is one less process on Unix platforms (i.e. no shared-manager process). choroba, please know that it is okay to reject this. I made a fork simply to test pm-cb-g using MCE::Child and MCE::Channel for my own validation.


I released MCE 1.841 (includes MCE::Channel and MCE::Child) and MCE::Shared 1.841. During development, ran pm-cb-g using MCE::Child and MCE::Channel on my laptop (macOS) including a Windows 7 VM. Here is the diff output (long form).

diff -aur pm-cb-master/lib/PM/CB/ pm-cb-master2/lib/PM/CB/Co --- pm-cb-master/lib/PM/CB/ 2019-06-08 16:07:06.000000000 + -0400 +++ pm-cb-master2/lib/PM/CB/ 2019-07-10 22:40:18.00000000 +0 -0400 @@ -15,6 +15,9 @@ sub start_comm { my ($self) = @_; $self->{communicate_t} = $self->{worker_class}->create(sub { + if ($INC{''}) { + $SIG{QUIT} = sub { threads->exit }; + } my $communication = PM::CB::Communication->new({ to_gui => $self->{to_gui}, from_gui => $self->{to_comm}, @@ -42,9 +45,11 @@ $self->{to_comm}->enqueue(['url']); }; } - $self->{to_comm}->insert(0, ['quit']); - $self->{communicate_t}->join; - $self->{to_gui}->insert(0, ['quit']); + if ($^O ne 'MSWin32' || ! $INC{'MCE/'}) { + $self->{communicate_t}->kill('QUIT'); + $self->{communicate_t}->join; + } + $self->{to_gui}->enqueue(['quit']); } diff -aur pm-cb-master/lib/PM/CB/ pm-cb-master2/lib/PM/CB/ --- pm-cb-master/lib/PM/CB/ 2019-06-08 16:07:06.000000000 -04 +00 +++ pm-cb-master2/lib/PM/CB/ 2019-07-06 15:17:07.000000000 -0 +400 @@ -673,7 +673,7 @@ sub quit { my ($self) = @_; print STDERR "Quitting...\n"; - $self->{to_control}->insert(0, ['quit']); + $self->{to_control}->enqueue(['quit']); } diff -aur pm-cb-master/pm-cb-g pm-cb-master2/pm-cb-g --- pm-cb-master/pm-cb-g 2019-06-08 16:07:06.000000000 -0400 +++ pm-cb-master2/pm-cb-g 2019-07-06 15:11:58.000000000 -0400 @@ -55,15 +55,15 @@ } -use if $mce => 'MCE::Hobo'; -use if $mce => 'MCE::Shared'; +use if $mce => 'MCE::Child'; +use if $mce => 'MCE::Channel'; use if ! $mce => threads => (stack_size => 2 ** $stack_size); use if ! $mce => 'Thread::Queue'; my ($queue_class, $queue_constructor, $worker_class) = $mce - ? ('MCE::Shared', 'queue', 'MCE::Hobo') + ? ('MCE::Channel', 'new', 'MCE::Child') : ('Thread::Queue', 'new', 'threads'); my ($to_gui, $to_comm, $to_control)

The ->insert method is lacking in MCE::Channel due to an edge case so not implemented. Signaling workers is another way and works well using threads on Windows, MCE::Hobo or MCE::Child on Unix. Threads requires one to define the signal handler whereas MCE::Hobo and MCE::Child already configure $SIG{QUIT} to terminate.

Update: I needed to enclose two lines above inside an if statement due to signal handling delayed for child processes (emulated) on the Windows platform. Perhaps the Tk loop is preventlng signaling from happening, am not sure. Tested the update on macOS and WIndows with and without the -m command-line switch.

+ if ( $^O ne 'MSWin32' || ! $INC{'MCE/'} ) { + $self->{communicate_t}->kill('QUIT'); + $self->{communicate_t}->join; + }

Q. Why does MCE::Channel lack the insert method?

A. Unlike MCE::Shared->queue, there is no manager process involvement in MCE::Channel. Thus, a worker holding a read lock on an empty channel (awaiting next item) makes it impossible for another worker inserting. Enqueue involves write lock only, but insert requires both read and write locks.

Regards, Mario

Replies are listed 'Best First'.
Re^5: A new CB reader
by marioroy (Vicar) on Jul 11, 2019 at 01:32 UTC

    Here is the smaller diff output (short form) for using MCE::Child and MCE::Channel. There is no involvment of the separate shared-manager process or thread due to not using MCE::Shared.

    diff -r pm-cb-master/lib/PM/CB/ pm-cb-master2/lib/PM/CB/Cont 17a18,20 > if ($INC{''}) { > $SIG{QUIT} = sub { threads->exit }; > } 45,47c48,52 < $self->{to_comm}->insert(0, ['quit']); < $self->{communicate_t}->join; < $self->{to_gui}->insert(0, ['quit']); --- > if ($^O ne 'MSWin32' || ! $INC{'MCE/'}) { > $self->{communicate_t}->kill('QUIT'); > $self->{communicate_t}->join; > } > $self->{to_gui}->enqueue(['quit']); diff -r pm-cb-master/lib/PM/CB/ pm-cb-master2/lib/PM/CB/ 676c676 < $self->{to_control}->insert(0, ['quit']); --- > $self->{to_control}->enqueue(['quit']); diff -r pm-cb-master/pm-cb-g pm-cb-master2/pm-cb-g 58,59c58,59 < use if $mce => 'MCE::Hobo'; < use if $mce => 'MCE::Shared'; --- > use if $mce => 'MCE::Child'; > use if $mce => 'MCE::Channel'; 66c66 < ? ('MCE::Shared', 'queue', 'MCE::Hobo') --- > ? ('MCE::Channel', 'new', 'MCE::Child')

    Regards, Mario

      All of your changes have been merged to my fork of pm-cb, which supports config files ($HOME/pm-cb.rc, $HOME/.pm-cbrc and/or $HOME/.config/pm-cb)

      Enjoy, Have FUN! H.Merijn
        Thank you! I tried to integrate the changes in my way. Please verify it works on MSWin, too. Is the QUIT signal still needed? Using MCE::Child makes quitting a bit slower, as all the messages in the queue must be processed first, but otherwise, it should work the same way as the other options.

        map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]

        Hi Tux,

        In truth, I never imagined for pm-cb-g to support both the -m and -M options. The reason I tested pm-cb-g using MCE::Child was for testing the 3 together Tk, MCE::Child, and MCE::Channel on Windows, Linux, and macOS.

        Using MCE::Hobo / MCE::Shared. Notice the extra shared-manager process.

        $ ps -e | grep perl 68028 ttys001 0:00.32 perl -I../test_mce pm-cb-g -m # main process 68029 ttys001 0:00.01 perl -I../test_mce pm-cb-g -m # shared mgr 68030 ttys001 0:00.01 perl -I../test_mce pm-cb-g -m # control 68031 ttys001 0:00.09 perl -I../test_mce pm-cb-g -m # communicate

        Using MCE::Child / MCE::Channel.

        $ ps -e | grep perl 68035 ttys001 0:00.29 perl -I../test_mce pm-cb-g -M # main process 68036 ttys001 0:00.01 perl -I../test_mce pm-cb-g -M # control 68037 ttys001 0:00.09 perl -I../test_mce pm-cb-g -M # communicate


        Q. Why was MCE::Hobo created?

        A. At the time, I wanted a threads-like parallel module to be included with MCE::Shared. It was also helpful for intense testing of MCE::Shared.

        Q. Why did I create MCE::Child years later?

        A. Well, MCE::Channel is new and included with MCE 1.841. I needed something to complement it including running on Perl 5.8. So, I tried making another threads-like parallel module. MCE::Child uses a common MCE::Channel object for IPC versus IPC handled by the shared-manager process. After several attempts, got it to work including running on the Windows platform.

        Q. Which one to choose?

        A. MCE::Child is lighter versus MCE::Hobo because not involving the shared-manager process. However, for apps that have shared variables (i.e. shared array, hash, scalar, etc.), then continue using MCE::Hobo. It's a moot point if the shared-manger is already running.

        Q. What about MCE::Channel->new versus MCE::Shared->queue?

        A. MCE::Channel involves no manager process and behaves very much like a pipe. It eliminates any concerns for Producer(s) running faster than Consumers. For MCE::Shared, items appended to a shared queue are sent to the shared-manager process where the data resides (i.e. in memory). This means the possibility for Producer(s) to run faster than Consumers. But no worries, MCE::Shared::Queue includes the await method for throttling if ever needed.

        Regards, Mario

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://11102656]
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (3)
As of 2019-11-17 09:56 GMT
Find Nodes?
    Voting Booth?
    Strict and warnings: which comes first?

    Results (86 votes). Check out past polls.