http://www.perlmonks.org?node_id=957906


in reply to Testing methodology

Why would you not use Thread::Conveyor? As for testing your module, adapting Conveyor.t would at least be a place to start.

Good Day,
    Dean

Replies are listed 'Best First'.
Re^2: Testing methodology
by BrowserUk (Pope) on Mar 05, 2012 at 14:36 UTC
    Why would you not use Thread::Conveyor?

    Because I know better. Why are you suggesting it when you obviously do not?

    I was asked to update this to explain my reasoning, so here it is:

    Thread::Conveyor doesn't work. It isn't thread-safe.

    Don't believe me, try it for yourself! I'll even provide the test script for you:

    #! perl -slw use strict; use threads ( stack_size => 4096 ); use Thread::Conveyor; my $belt = Thread::Conveyor->new( { maxboxes => 50, minboxes => 25, optimize => 'memory', # or 'cpu' }); async{ print while defined( $_ = $belt->take ); }->detach; $belt->put( $_ ) for 1 .. 10; $belt->put( undef );

    Nine times out of ten this will segfault with;

    Problem signature: Problem Event Name: APPCRASH Application Name: perl.exe Application Version: 5.10.1.1007 Application Timestamp: 4b60ba96 Fault Module Name: perl510.dll Fault Module Version: 5.10.1.1007 Fault Module Timestamp: 4b60ba95 Exception Code: c0000005 Exception Offset: 000000000006c6f8 OS Version: 6.0.6001.2.1.0.768.3 Locale ID: 2057 Additional Information 1: 90e0 Additional Information 2: e939a93db866b76af40148e39e07fd0d Additional Information 3: 85c4 Additional Information 4: 6c632c487ffa8e4b9b7137dbbbe72313

    On the 10th occasion it will emit the following before hanging:

    C:\test>t-TCcrap 1 2 3 4 5 6 7 8 Thread 1 terminated abnormally: Can't use an undefined value as an ARR +AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem +and from offset 1939 for 176 bytes) line 75. Terminating on signal SIGINT(2)

    And if you trace the run you get:

      Thank you very much for updating and giving your reason. I do in fact use Thread::Conveyor and had never run in to the segfault issue.

      Upon investigation, I only get the segfault when I set the stack size globally (when I load the threads module or via threads->set_stack_size(). I have no problems if I set the stack size at thread creation time (below). This explains why I have never run into trouble. Clearly, there is some problem with Thread::Conveyor and you have convinced me that there might be some reason to avoid it. Determining whether the problem is fixable is probably beyond me.

      #! perl -slw # This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi use strict; use threads; use Thread::Conveyor; my $belt = Thread::Conveyor->new( { maxboxes => 50, minboxes => 25, optimize => 'memory', # or 'cpu' }); threads->new(sub { print while defined( $_ = $belt->take ); }, { stack_size => 4096 })->detach; $belt->put( $_ ) for 1 .. 10; $belt->put( undef );

      Good Day,
          Dean

        FWIW: Here, the crash seems to be unrelated to the stack_size parameter, regardless of where it is applied. It seems to be entirely due to global clean up.

        That said, even if I defer global clean up, it still has severe problems.

        • It will frequently dq items twice or more.
        • Try to access non-existent stack items. And then hang.
        • If you try to use more than one consumer and one producer per queue, it just hangs.
        • If you make the queue too big, it hangs.
        • Under the rarefied conditions that it actually completes -- except for dodging the global clean-up issue -- it is very slow.

        Here's my test code and some typical results (Terminated on sigint means it hung):

        #! perl -slw use strict; use threads; use threads::shared; use Time::HiRes qw[ time sleep ]; use Thread::Conveyor; our $O //= 'memory'; our $N //= 1e4; our $T //= 4; ++$T; $T &= ~1; our $SIZE //= 50; my $Q1_n = Thread::Conveyor->new( { maxboxes => $SIZE, optimize => $O +}); my $Qn_n = Thread::Conveyor->new( { maxboxes => $SIZE, optimize => $O +}); my $Qn_1 = Thread::Conveyor->new( { maxboxes => $SIZE, optimize => $O +}); my $done1 :shared = 0; my $done2 :shared = 0; my @t1 = map async( sub{ $Qn_n->put( $_ ) while defined( $_ = $Q1_n->t +ake ); lock $done1; ++$done1; } ), 1 .. $T/2; my @t2 = map async( sub{ $Qn_1->put( $_ ) while defined( $_ = $Qn_n->t +ake ); lock $done2; ++$done2; } ), 1 .. $T/2; my $bits :shared = chr(0); $bits x= $N/ 8 + 1; my $t = async{ while( defined( $_ = $Qn_1->take ) ) { warn "value duplicated" if vec( $bits, $_, 1 ); vec( $bits, $_, 1 ) = 1; } }; my $start = time; $Q1_n->put( $_ ) for 1 .. $N; $Q1_n->put( (undef) x ($T/2) ); sleep 0.01 while $done1 < @t1; $Qn_n->put( (undef) x ($T/2) ); sleep 0.01 while $done2 < @t2; $Qn_1->put( undef ); sleep 0.01 until $t->is_joinable; my $stop = time; my $b = unpack '%32b*', $bits; die "NOK: $b : \n" unless $b == $N; printf "$N items by $T threads via three Qs size $SIZE in %.6f seconds +\n", $stop - $start; sleep 10; __END__ C:\test>perl async\Q.pm -N=1e4 -T=2 -SIZE=40 1e4 items by 2 threads via three Qs size 40 in 5.458000 seconds C:\test>t-TCcrap -N=100 -T=2 -SIZE=40 100 items by 2 threads via three Qs size 40 in 0.148909 seconds C:\test>t-TCcrap -N=1e4 -T=2 -SIZE=40 1e4 items by 2 threads via three Qs size 40 in 11.035008 seconds C:\test>t-TCcrap -N=1e5 -T=2 -SIZE=40 value duplicated at C:\test\t-TCcrap.pl line 26. value duplicated at C:\test\t-TCcrap.pl line 26. Thread 1 terminated abnormally: Can't use an undefined value as an ARR +AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem +and from offset 1939 for 176 bytes) line 75. Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=100 -T=2 -SIZE=4 100 items by 2 threads via three Qs size 4 in 0.148934 seconds Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=100 -T=4 -SIZE=4 Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=100 -T=4 -SIZE=4 Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=100 -T=3 -SIZE=4 Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=100 -T=3 -SIZE=40 Thread 1 terminated abnormally: Can't use an undefined value as an ARR +AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem +and from offset 1939 for 176 bytes) line 75. Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=100 -T=3 -SIZE=40 value duplicated at C:\test\t-TCcrap.pl line 26. value duplicated at C:\test\t-TCcrap.pl line 26. Terminating on signal SIGINT(2) C:\test>t-TCcrap -N=1e5 -T=2 -SIZE=40 value duplicated at C:\test\t-TCcrap.pl line 26. value duplicated at C:\test\t-TCcrap.pl line 26. Thread 1 terminated abnormally: Can't use an undefined value as an ARR +AY reference at C:/Perl64/site/lib/Thread/Tie/Array.pm (loaded on dem +and from offset 1939 for 176 bytes) line 75. Terminating on signal SIGINT(2)

        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?