Why isn't this code thread-safe? (Is "require" thread-safe??)

[vr]

I was running a simple task, almost one-shot, throw-away script, therefore using basic tools. Because there were thousands of files to process, I decided to parallelize. With small test-suite, and "no-op", SSCCE code, sometimes output is OK:

use strict;
use warnings;
use feature 'say';
use threads;
use Thread::Queue;
use CAM::PDF;
use File::Find;

my $q = Thread::Queue-> new;

my @gang = map async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        say threads-> tid, ' ', $f;
        my $pdf = CAM::PDF-> new( $f ) or die;
    }
}), 1 .. 2;

find( sub {
    -f and /\.pdf$/i and $q-> enqueue( $File::Find::name )
}, './1' );

$q-> end;
$_-> join for @gang;

__END__

1 ./1/1/106/10627.pdf
2 ./1/1/107/10703.pdf
2 ./1/1/186/18673.pdf
1 ./1/1/209/20946.pdf
2 ./1/1/26/2656.pdf
1 ./1/1/33/3384.pdf
2 ./1/1/57/5742.pdf
1 ./1/1/58/5869.pdf
2 ./1/1/63/6395.pdf
1 ./1/1/70/7099.pdf
1 ./1/1/74/7466.pdf

But sometimes not (example 1, one worker dead):

1 ./1/1/106/10627.pdf
2 ./1/1/107/10703.pdf
Thread 2 terminated abnormally: *****Undefined subroutine &Compress::Z
+lib::Parse
Parameters called at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/lib/Co
+mpress/Zli
b.pm line 366.
1 ./1/1/186/18673.pdf
1 ./1/1/209/20946.pdf
1 ./1/1/26/2656.pdf
1 ./1/1/33/3384.pdf
1 ./1/1/57/5742.pdf
1 ./1/1/58/5869.pdf
1 ./1/1/63/6395.pdf
1 ./1/1/70/7099.pdf
1 ./1/1/74/7466.pdf

Example 2 (both workers dead, but for different reason):

1 ./1/1/106/10627.pdf
2 ./1/1/107/10703.pdf
Thread 2 terminated abnormally: *****Global symbol "@ISA" requires exp
+licit pack
age name (did you forget to declare "my @ISA"?) at C:/strawberry-perl-
+5.28.0.1-3
2bit-PDL/perl/site/lib/Text/PDF/Filter.pm line 342.
Global symbol "@basedict" requires explicit package name (did you forg
+et to decl
are "my @basedict"?) at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/sit
+e/lib/Text
/PDF/Filter.pm line 343.
Global symbol "@basedict" requires explicit package name (did you forg
+et to decl
are "my @basedict"?) at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/sit
+e/lib/Text
/PDF/Filter.pm line 351.
Global symbol "@basedict" requires explicit package name (did you forg
+et to decl
are "my @basedict"?) at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/sit
+e/lib/Text
/PDF/Filter.pm line 374.
Compilation failed in require at C:/strawberry-perl-5.28.0.1-32bit-PDL
+/perl/site
/lib/CAM/PDF.pm line 5608.
Thread 1 terminated abnormally: *****Global symbol "@ISA" requires exp
+licit pack
age name (did you forget to declare "my @ISA"?) at C:/strawberry-perl-
+5.28.0.1-3
2bit-PDL/perl/site/lib/Text/PDF/Filter.pm line 342.
Global symbol "@basedict" requires explicit package name (did you forg
+et to decl
are "my @basedict"?) at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/sit
+e/lib/Text
/PDF/Filter.pm line 343.
Global symbol "@basedict" requires explicit package name (did you forg
+et to decl
are "my @basedict"?) at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/sit
+e/lib/Text
/PDF/Filter.pm line 351.
Global symbol "@basedict" requires explicit package name (did you forg
+et to decl
are "my @basedict"?) at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/sit
+e/lib/Text
/PDF/Filter.pm line 374.
Compilation failed in require at C:/strawberry-perl-5.28.0.1-32bit-PDL
+/perl/site
/lib/CAM/PDF.pm line 5608.

Actually, CAM::PDF, "as is", is coded to issue a single warning (large source file!), but with filter undefined it becomes somewhat broken and useless and floods terminal with further thousands of warnings, therefore I prepended that line with

die '*****' . $@;

so the output is as shown above. My impression is that threads are trying to read the same source files -- CAM::PDF requires Text::PDF::Filter, which requires Compress::Zlib, and hence some sort of race condition happens and failed (partial) reading from file.

Is that even possible? I thought that files can be opened for reading safely by different processes, and OS would "arbitrate" "parallel" access to them. Is it not the case in general, or with require only?

If it's not the case, then is it a common knowledge (which I missed) that main thread should take care to "pre-require" all modules possibly needed by several workers before spawning them?

(Note, if someone wants to run tests: PDFs are of "compressed xref table" variety (they are client's files I won't share), and with other (simple xref table) files the sub containing line 5608 won't be called, i.e. Text::PDF::Filter won't be required, on simply reading a file).

[BrowserUk]

The problem is almost certainly that CAM::PDF or one of its many dependencies isn't thread safe. I vaguely recall finding that one of the GZIP modules used some global state internally at the C/XS level. There are ways around it (using threads), but they require knowledge and testing I'm no longer in a position to supply, but in any case there is a better way for this type of application: marioroy's MCE.

He has posted dozens of well written and tested examples of exactly this type of IO multi-tasking problem, and has proven himself willing and able to actively support those using his code.

And finally, even my best attempts at IO multitasking using threads never came close to achieving the same level of throughput and performance that he achieved with his early versions; and his latest stuff is even more efficient.

[vr]

I vaguely recall finding that one of the GZIP modules used some global state

Thanks for answer, I've reduced the problem to Compress::Zlib:

use strict;
use warnings;
use feature 'say';
use threads;
use Thread::Queue;

my $q = Thread::Queue-> new;

my @gang = map async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        require Compress::Zlib;
        say threads-> tid;
    }
}), 1 .. 4;

select( undef, undef, undef, 0.1 ) or $q-> enqueue( $_ ) for 1 .. 4;
$q-> end;
$_-> join for @gang;

With small (or none) delays between threads "requiring" Compress::Zlib, errors are happenning:

D:\>perl test181110.pl
2
3
1
4

D:\>perl test181110.pl
2
4
3
1

D:\>perl test181110.pl
3
2
1
4

D:\>perl test181110.pl
String found where operator expected at C:/strawberry-perl-5.28.0.1-32
+bit-PDL/pe
rl/lib/IO/Compress/Base/Common.pm line 514, near "croak "$sub: $p->[Ix
+Error]""
        (Do you need to predeclare croak?)
Thread 3 terminated abnormally: syntax error at C:/strawberry-perl-5.2
+8.0.1-32bi
t-PDL/perl/lib/IO/Compress/Base/Common.pm line 514, near "croak "$sub:
+ $p->[IxEr
ror]""
BEGIN not safe after errors--compilation aborted at C:/strawberry-perl
+-5.28.0.1-
32bit-PDL/perl/lib/IO/Compress/Base/Common.pm line 520.
Compilation failed in require at C:/strawberry-perl-5.28.0.1-32bit-PDL
+/perl/lib/
Compress/Zlib.pm line 10.
BEGIN failed--compilation aborted at C:/strawberry-perl-5.28.0.1-32bit
+-PDL/perl/
lib/Compress/Zlib.pm line 10.
Compilation failed in require at test181110.pl line 11.
1
4
2

D:\>perl test181110.pl
Thread 3 terminated abnormally: Bareword "HIGH" not allowed while "str
+ict subs"
in use at C:/strawberry-perl-5.28.0.1-32bit-PDL/perl/lib/IO/Compress/B
+ase/Common
.pm line 868.
BEGIN not safe after errors--compilation aborted at C:/strawberry-perl
+-5.28.0.1-
32bit-PDL/perl/lib/IO/Compress/Base/Common.pm line 1038.
Compilation failed in require at C:/strawberry-perl-5.28.0.1-32bit-PDL
+/perl/lib/
Compress/Zlib.pm line 10.
BEGIN failed--compilation aborted at C:/strawberry-perl-5.28.0.1-32bit
+-PDL/perl/
lib/Compress/Zlib.pm line 10.
Compilation failed in require at test181110.pl line 11.
1
2
4

D:\>perl test181110.pl
1
3
1
2

D:\>perl test181110.pl
1
3
2
4

But if delay is increased to 0.3 script is run hundreds of times, with batch command, w/o errors. Instead of delays (value relevant to one PC only), module should be "used" in main thread, of course.

I don't understand why text of errors indicates incomplete or "noisy" reading (parsing) of Perl source files, as if random lines are just omitted. And in OP, same "broken source" happens to rather simple Text::PDF::Filter. As I see, there's relatively complex chain of require's in Text::PDF::Filter, to Compress::Zlib and further to parts of that distribution. Maybe there is same "long chain of requires" in some other distributions, to check with script above?

[BrowserUk]

I think moving to requireing the module tree in the thread subs is probably compounding the problems.

Imagine one thread gets a timeslice, gets part way through loading the module tree and then gets interrupted and another thread starts its attempt to load those same modules. If one of them has global state that is only used during loading, that interruption may leave it in an undefined state and the second thread inherits that state at the C level.

If you really want to go that route, you should consider wrapping the require in a critical section to ensure that no two threads can be attempting to load the module tree concurrently.

[marioroy]

Hello vr,

One can try loading IO::Handle before spawning workers. That alone is helpful for increasing reliability for modules that involve IO::*. I was able to reproduce threads failing but not after adding the IO::Handle line. Even tested with 100 threads without a delay between them.

use warnings;
use feature 'say';
use threads;
use Thread::Queue;

use IO::Handle;  # <-- important

my $q = Thread::Queue-> new;

my @gang = map async( sub {
    while ( defined( my $f = $q-> dequeue )) {
        require Compress::Zlib;
        say threads-> tid;
    }
}), 1 .. 8;

$q-> enqueue( $_ ) for 1 .. 8;
$q-> end;

$_-> join for @gang;

Kind regards, Mario