Re^2: Testing methodology

[BrowserUk]

Why would you not use Thread::Conveyor?

Because I know better. Why are you suggesting it when you obviously do not?

I was asked to update this to explain my reasoning, so here it is:

Thread::Conveyor doesn't work. It isn't thread-safe.

Don't believe me, try it for yourself! I'll even provide the test script for you:

#! perl -slw
use strict;
use threads ( stack_size => 4096 );
use Thread::Conveyor;

my $belt = Thread::Conveyor->new( {
    maxboxes => 50,
    minboxes => 25,
    optimize => 'memory', # or 'cpu'
});

async{
    print while defined( $_ = $belt->take );
}->detach;

$belt->put( $_ ) for 1 .. 10;
$belt->put( undef );

Nine times out of ten this will segfault with;

Problem signature:
  Problem Event Name:    APPCRASH
  Application Name:    perl.exe
  Application Version:    5.10.1.1007
  Application Timestamp:    4b60ba96
  Fault Module Name:    perl510.dll
  Fault Module Version:    5.10.1.1007
  Fault Module Timestamp:    4b60ba95
  Exception Code:    c0000005
  Exception Offset:    000000000006c6f8
  OS Version:    6.0.6001.2.1.0.768.3
  Locale ID:    2057
  Additional Information 1:    90e0
  Additional Information 2:    e939a93db866b76af40148e39e07fd0d
  Additional Information 3:    85c4
  Additional Information 4:    6c632c487ffa8e4b9b7137dbbbe72313

On the 10th occasion it will emit the following before hanging:

C:\test>t-TCcrap
1
2
3
4
5
6
7
8
Thread 1 terminated abnormally: Can't use an undefined value as an ARR
+AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem
+and from offset 1939 for 176 bytes) line 75.
Terminating on signal SIGINT(2)

And if you trace the run you get:

<Reveal this spoiler or all in this thread>

[duelafn]

Thank you very much for updating and giving your reason. I do in fact use Thread::Conveyor and had never run in to the segfault issue.

Upon investigation, I only get the segfault when I set the stack size globally (when I load the threads module or via threads->set_stack_size(). I have no problems if I set the stack size at thread creation time (below). This explains why I have never run into trouble. Clearly, there is some problem with Thread::Conveyor and you have convinced me that there might be some reason to avoid it. Determining whether the problem is fixable is probably beyond me.

#! perl -slw
# This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi
use strict;
use threads;
use Thread::Conveyor;

my $belt = Thread::Conveyor->new( {
    maxboxes => 50,
    minboxes => 25,
    optimize => 'memory', # or 'cpu'
});

threads->new(sub {
    print while defined( $_ = $belt->take );
},
{ stack_size => 4096 })->detach;

$belt->put( $_ ) for 1 .. 10;
$belt->put( undef );

[BrowserUk]

FWIW: Here, the crash seems to be unrelated to the stack_size parameter, regardless of where it is applied. It seems to be entirely due to global clean up.

That said, even if I defer global clean up, it still has severe problems.

• It will frequently dq items twice or more.
• Try to access non-existent stack items. And then hang.
• If you try to use more than one consumer and one producer per queue, it just hangs.
• If you make the queue too big, it hangs.
• Under the rarefied conditions that it actually completes -- except for dodging the global clean-up issue -- it is very slow.

Here's my test code and some typical results (Terminated on sigint means it hung):

#! perl -slw
use strict;
use threads;
use threads::shared;
use Time::HiRes qw[ time sleep ];
use Thread::Conveyor;

our $O //= 'memory';
our $N //= 1e4;
our $T //= 4; ++$T; $T &= ~1;
our $SIZE //= 50;

my $Q1_n = Thread::Conveyor->new( { maxboxes => $SIZE, optimize => $O 
+});
my $Qn_n = Thread::Conveyor->new( { maxboxes => $SIZE, optimize => $O 
+});
my $Qn_1 = Thread::Conveyor->new( { maxboxes => $SIZE, optimize => $O 
+});

my $done1 :shared = 0;
my $done2 :shared = 0;

my @t1 = map async( sub{ $Qn_n->put( $_ ) while defined( $_ = $Q1_n->t
+ake ); lock $done1; ++$done1; } ), 1 .. $T/2;
my @t2 = map async( sub{ $Qn_1->put( $_ ) while defined( $_ = $Qn_n->t
+ake ); lock $done2; ++$done2; } ), 1 .. $T/2;

my $bits :shared = chr(0); $bits x= $N/ 8 + 1;
my $t = async{
    while( defined( $_ = $Qn_1->take ) ) {
        warn "value duplicated" if vec( $bits, $_, 1 );
        vec( $bits, $_, 1 ) = 1;
    }
};

my $start = time;

$Q1_n->put( $_ ) for 1 .. $N;
$Q1_n->put( (undef) x ($T/2) ); sleep 0.01 while $done1 < @t1;
$Qn_n->put( (undef) x ($T/2) );  sleep 0.01 while $done2 < @t2;
$Qn_1->put( undef );             sleep 0.01 until $t->is_joinable;

my $stop = time;

my $b = unpack '%32b*', $bits;
die "NOK: $b : \n" unless $b == $N;

printf "$N items by $T threads via three Qs size $SIZE in %.6f seconds
+\n",
    $stop - $start;

sleep 10;

__END__
C:\test>perl async\Q.pm -N=1e4 -T=2 -SIZE=40
1e4 items by 2 threads via three Qs size 40 in 5.458000 seconds

C:\test>t-TCcrap -N=100 -T=2 -SIZE=40
100 items by 2 threads via three Qs size 40 in 0.148909 seconds

C:\test>t-TCcrap -N=1e4 -T=2 -SIZE=40
1e4 items by 2 threads via three Qs size 40 in 11.035008 seconds

C:\test>t-TCcrap -N=1e5 -T=2 -SIZE=40
value duplicated at C:\test\t-TCcrap.pl line 26.
value duplicated at C:\test\t-TCcrap.pl line 26.
Thread 1 terminated abnormally: Can't use an undefined value as an ARR
+AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem
+and from offset 1939 for 176 bytes) line 75.
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=100 -T=2 -SIZE=4
100 items by 2 threads via three Qs size 4 in 0.148934 seconds
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=100 -T=4 -SIZE=4
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=100 -T=4 -SIZE=4
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=100 -T=3 -SIZE=4
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=100 -T=3 -SIZE=40
Thread 1 terminated abnormally: Can't use an undefined value as an ARR
+AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem
+and from offset 1939 for 176 bytes) line 75.
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=100 -T=3 -SIZE=40
value duplicated at C:\test\t-TCcrap.pl line 26.
value duplicated at C:\test\t-TCcrap.pl line 26.
Terminating on signal SIGINT(2)

C:\test>t-TCcrap -N=1e5 -T=2 -SIZE=40
value duplicated at C:\test\t-TCcrap.pl line 26.
value duplicated at C:\test\t-TCcrap.pl line 26.
Thread 1 terminated abnormally: Can't use an undefined value as an ARR
+AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem
+and from offset 1939 for 176 bytes) line 75.
Terminating on signal SIGINT(2)