Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Re^3: Why isn't this code thread-safe? (MCE!)

by marioroy (Priest)
on Nov 19, 2018 at 22:06 UTC ( #1226032=note: print w/replies, xml ) Need Help??


in reply to Re^2: Why isn't this code thread-safe? (MCE!)
in thread Why isn't this code thread-safe? (Is "require" thread-safe??)

Hello vr,

One can try loading IO::Handle before spawning workers. That alone is helpful for increasing reliability for modules that involve IO::*. I was able to reproduce threads failing but not after adding the IO::Handle line. Even tested with 100 threads without a delay between them.

use warnings; use feature 'say'; use threads; use Thread::Queue; use IO::Handle; # <-- important my $q = Thread::Queue-> new; my @gang = map async( sub { while ( defined( my $f = $q-> dequeue )) { require Compress::Zlib; say threads-> tid; } }), 1 .. 8; $q-> enqueue( $_ ) for 1 .. 8; $q-> end; $_-> join for @gang;

Kind regards, Mario

Replies are listed 'Best First'.
Re^4: Why isn't this code thread-safe? (MCE!)
by marioroy (Priest) on Nov 19, 2018 at 22:25 UTC

    Hi again,

    Here is the same thing using MCE::Hobo. Similar code, but processes instead.

    use warnings; use feature 'say'; use MCE::Hobo; use MCE::Shared; # use IO::Handle; # <-- loaded automatically by MCE and MCE::Shared:: +Server my $q = MCE::Shared-> queue; my @gang = map mce_async( sub { while ( defined( my $f = $q-> dequeue )) { require Compress::Zlib; say MCE::Hobo-> tid; } }), 1 .. 100; $q-> enqueue( $_ ) for 1 .. 100; $q-> end; $_-> join for @gang;

    For modules not multi-process safe, another thing one can do on Unix platforms is having MCE::Hobo default to posix_exit to avoid all END and destructor processing.

    use warnings; use feature 'say'; use MCE::Hobo; use MCE::Shared; # use IO::Handle; # <-- loaded automatically by MCE before spawning MCE::Hobo->init( posix_exit => 1 ); my $q = MCE::Shared-> queue; my @gang = map mce_async( sub { while ( defined( my $f = $q-> dequeue )) { require Compress::Zlib; say MCE::Hobo-> tid; } }), 1 .. 8; $q-> enqueue( $_ ) for 1 .. 8; $q-> end; $_-> join for @gang;

    Note that the posix_exit option is not recommended if constructing an object inside the worker involving a temp file. In that case one may want the worker to exit normally. Anyway, the posix_exit option is there if needed as a last resort.

    Taken from the MCE::Hobo manual: Set posix_exit to avoid all END and destructor processing. Constructing MCE::Hobo inside a thread implies 1 or if present CGI, FCGI, Coro, Curses, Gearman::Util, Gearman::XS, LWP::UserAgent, Mojo::IOLoop, Prima, STFL, Tk, Wx, or Win32::GUI.

    A lot of modules are not multi-process safe and the reason for setting to 1 automatically. Btw, Prima is now multi-process safe recently.

    Kind regards, Mario

Re^4: Why isn't this code thread-safe? (MCE!)
by vr (Chaplain) on Nov 20, 2018 at 16:48 UTC

    So, the problem is not about Compress::Zlib (or poor Text::PDF::Filter... did we torture it beyond convalescing? oops, sorry). The minimal script is then:

    use strict; use warnings; use feature 'say'; use threads; use Thread::Queue; my $q = Thread::Queue-> new; my @gang = map async( sub { while ( defined( my $f = $q-> dequeue )) { require IO::Handle; say threads-> tid; } }), 1 .. 8; $q-> enqueue( $_ ) for 1 .. @gang; $q-> end; $_-> join for @gang;

    which sometimes (often, both Windows and Linux) fails randomly. Further, replacing the require line with

    use XSLoader; XSLoader::load 'IO';

    sometimes there are strange warnings, like:

    Constant subroutine SEEK_END redefined at threads.pl line 4294967295

    Maybe this line number ("broken source"?) is important, maybe not. Until (if ever) the issue is fixed, maybe it would be good idea to advise (documentation, i.e.) to use IO:Handle in any threaded code? Won't hurt...

    As to issue of "multi-process safe modules", I can't get same problem if above script is simply re-written using fork (Linux), perhaps it's unrelated issue?

    And, certainly, thanks for showing that even a throw-away, few dozen lines script is as easily written using MCE, as "vanilla Perl", except it "just works", because author has put his knowledge and effort to take care of possible bugs. Thanks, Mario :)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1226032]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (2)
As of 2019-03-23 06:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How do you Carpe diem?





    Results (114 votes). Check out past polls.

    Notices?