Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling
 
PerlMonks  

Perl 5.8.2 thread is worse - more findings from today

by pg (Canon)
on Dec 28, 2003 at 05:43 UTC ( #317281=perlmeditation: print w/ replies, xml ) Need Help??

After I posted Perl 5.8.2 thread is much worse, I tried more, and more bugs are found.

This morning, I actually agreed with Aristotle in a private conversation that, it might be too strong for me to say .2 is worse, instead just say .2 is as problematic as .1

The impression I had earlier was that:

  • detach() is now unfortunately broken
  • however a possible workaround is to remove detach(). In .1, without detach, memory leaks quickly led to core dump. In .2 as the memory leak is removed, it is now safe without detach()

However today that script I posted in my previous post, core dumped twice, without detach(). Each time it survives 2 hours. As I monitoring, the memory usage increased slowly indicating a memory leak, but much tamed than before.

Now it is absolutely justified to say it is worse, as both alternatives are broken, with or without detach():

  • With detach(), core dumps right the way (which was not the case in .1)
  • Without detach(), survives for hours but not too long. (better than .1, but from a production system point view, surviving for 2 hours is the same as surviving for 10 seconds)

I have reported both problems to p5p, hope still can catch the code freeze for .3 (well obviously also hope there will be a fix before .3 time frame).

Comment on Perl 5.8.2 thread is worse - more findings from today
Re: Perl 5.8.2 thread is worse - more findings from today
by castaway (Parson) on Dec 28, 2003 at 13:05 UTC
    Im still curious as to exactly how its broken. I just tried some simple thread code using detach() and got no core dumps.

    Can you cut the code down to something that shows that its actually 'detach' thats breaking it, and why? (just removing/adding it might have some other effects).

    Maybe you could also find out where memory is leaking.. One of the nice things I noticed after upgrading to 5.8.2 was that my script (the im2 script, actually) stopped leaking memory.. in 5.8.0 and 5.8.1 it was getting to around 150mb before leveling off, now in 5.8.2 it can run for days at a time and only peak around 25mb.

    Without trying to find where the bugs are exactly, I doubt you will get much response from p5p, since they probably dont want to dig through your code, and wont get much out of a 'detach produces core dumps' error..

    C. still struggling to figure out if blessed objects 'work' .. currently they seem to be working, darn it..

      "Can you cut the code down to something that shows that its actually 'detach' thats breaking it"

      Okay, that's easy. It only took me 1 minute to come up with this code, and break (core dump) it: (I repeated 3 times to make sure it is really broken)

      use strict; use warnings; use threads; for (my $i = 1; $i < 1000; $i ++) { threads->create(\&abc)->detach(); } sub abc { }

      Update:

      It was a waste to create 1000 threads, and side tracks people's focus. This is much more simple and clear:

      use strict; use warnings; use threads; for (my $i = 1; $i < 3; $i ++) { threads->create(\&abc)->detach(); } <>; sub abc { }

      Update 2:

      The testing is done on win32, using .2. This was mentioned in the first stream of notes Perl 5.8.2 thread is much worse on .2 threads. Any way, I should have repeated this here earlier.

        Could you try it with this change (in pseudo diff format):
        sub abc { +undef }
        A bug (new to 5.8.2) causes problems with empty subroutines. Put the undef in there, and see if that helps. I've had to update several of my modules to get around this problem for threaded 5.8.2.

        I should have thought about this earlier. The whole thing sort of left my mind already... ;-) Good thing I document my changes... ;-)

        BTW, this should already be fixed for 5.8.3.

        Liz

        No core dump using that code on my perl 5.8.2:
        This is perl, v5.8.2 built for i686-linux-thread-multi
        
        Copyright 1987-2003, Larry Wall
        
        Perl may be copied only under the terms of either the Artistic License or the
        GNU General Public License, which may be found in the Perl 5 source kit.
        
        Complete documentation for Perl, including FAQ lists, should be found on
        this system using `man perl' or `perldoc perl'.  If you have access to the
        Internet, point your browser at http://www.perl.com/, the Perl Home Page.
        

        C.

        I can partly confirm the results with my perl 5.8.2 under Windows 2000 - some times, the snippet crashes with an access violation, but sometimes it dosen't. I could not safely reproduce the bug all times. This looks to me like a weird bug in Perl to me, some uninitialized pointer somewhere.

        Update: In my search for the cause, I found the following behaviour - the segfault appears within a spawned thread, as the following script exits even as the segfault messagebox is still on screen:

        use strict; use warnings; use threads; for (1..2) { threads->create(sub { undef })->detach(); }; sleep 1;
        Summary of my perl5 (revision 5 version 8 subversion 2) configuration: Platform: osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef usethreads=undef use5005threads=undef useithreads=define usemultip +licity=def ine useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -Gf -W3 -MD -DNDEBUG -O1 -DWIN32 -D_CON +SOLE -DNO_ STRICT -DHAVE_DES_FCRYPT -DPERL_IMPLICIT_CONTEXT -DPERL_IMPLICIT_SYS +-DUSE_PERL IO -DPERL_MSVCRT_READFIX', optimize='-MD -DNDEBUG -O1', cppflags='-DWIN32' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64 +', lseeksi ze=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -release -libpath:"d:\ +perl\5.8.2 \lib\CORE" -machine:x86' libpth=C:\PROGRA~1\MICROS~3\VC98\lib libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib + comdlg32 .lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uui +d.lib wsoc k32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msvcrt. +lib perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool +.lib comd lg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib + uuid.lib wsock32.lib mpr.lib winmm.lib version.lib odbc32.lib odbccp32.lib msv +crt.lib libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib gnulibc_version='undef' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -release -l +ibpath:"d: \perl\5.8.2\lib\CORE" -machine:x86' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL +_IMPLICIT_ CONTEXT PERL_IMPLICIT_SYS Built under MSWin32 Compiled at Nov 6 2003 12:48:55 @INC: D:/perl/5.8.2/lib D:/perl/site/5.8.2/lib .
        perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The $d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider ($c = $d->accept())->get_request(); $c->send_response( new #in the HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web
        If invoke perl interactively like so, no memory access violations.
        E:\>perl use strict; use warnings; use threads; for (my $i = 1; $i < 3; $i ++) { threads->create(\&abc)->detach(); } <>; sub abc { } __END__ E:\>perl -V Summary of my perl5 (revision 5 version 8 subversion 2) configuration: Platform: osname=MSWin32, osvers=4.0, archname=MSWin32-x86-multi-thread uname='' config_args='undef' hint=recommended, useposix=true, d_sigaction=undef usethreads=undef use5005threads=undef useithreads=define usemultip +licity=define useperlio=define d_sfio=undef uselargefiles=define usesocks=undef use64bitint=undef use64bitall=undef uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cl', ccflags ='-nologo -Gf -W3 -MD -DNDEBUG -O1 -DWIN32 -D_CON +SOLE -DNO_STRICT -DHAVE_DES_FCRYPT -DPERL_IMPLIC IT_CONTEXT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX', optimize='-MD -DNDEBUG -O1', cppflags='-DWIN32' ccversion='', gccversion='', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64 +', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='link', ldflags ='-nologo -nodefaultlib -release -libpath:"G:\ +perls\5.8.2\lib\CORE" -machine:x86' libpth=\lib \site\lib libs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib + comdlg32.lib advapi32.lib shell32.lib ole32.lib oleaut32.lib netapi32.lib uuid.lib wsock32.lib mpr.lib winmm.lib ver +sion.lib odbc32.lib odbccp32.lib msvcrt.lib perllibs= oldnames.lib kernel32.lib user32.lib gdi32.lib winspool +.lib comdlg32.lib advapi32.lib shell32.lib ole32. lib oleaut32.lib netapi32.lib uuid.lib wsock32.lib mpr.lib winmm.lib + version.lib odbc32.lib odbccp32.lib msvcrt.lib libc=msvcrt.lib, so=dll, useshrplib=yes, libperl=perl58.lib gnulibc_version='undef' Dynamic Linking: dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' ' cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -release -l +ibpath:"G:\perls\5.8.2\lib\CORE" -machine:x86' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL +_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS Built under MSWin32 Compiled at Nov 5 2003 21:13:21 %ENV: PERL5LIB="G:\perls\5.8.2\site\lib;" @INC: G:\perls\5.8.2\site\lib G:/perls/5.8.2/lib . E:\>
        update: ok, if I save it to a file, it happens, d'oh. The bugs appears in threads.dll according to mah debugger (Unhandled exception in perl.exe(THREADS.DLL): 0XC0000005 ). And apparently, line 152 in threads.xs is the culprit (        PerlMemShared_free(thread);)

        update: I should note that both the threads cause this violation, but that this is only apparent in the debug version (in the non-debug version it's death after the 1st), and that like corion, I can't reliably reproduce this (sometimes I have no problems running the snippet, sometimes I do).

        It appears that the interactive invocation can cause the error also.

        MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
        I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
        ** The third rule of perl club is a statement of fact: pod is sexy.

        under cygwin, which basicaly is win32 with a *nix layer this code core dump
        # echo|perl thisscript use strict; use warnings; use threads; for (my $i = 1; $i < 3; $i ++) { threads->create(\&abc)->detach(); } <>; sub abc { }
        while this one is OK
        #perl use strict; use warnings; use threads; for (my $i = 1; $i < 3; $i ++) { threads->create(\&abc)->detach(); } sleep 1; sub abc { }
        ... I've tried to mix case of empty sub and processing the <>.
        --
        dominix
Re: Perl 5.8.2 thread is worse - more findings from today
by liz (Monsignor) on Dec 28, 2003 at 14:20 UTC
    The perldelta.pod of 5.8.2 states:
    Updated Modules And Pragmata
    The following modules and pragmata have been updated since Perl 5.8.1:
           Devel::PPPort
           Digest::MD5
           I18N::LangTags
           libnet
           MIME::Base64
           Pod::Perldoc
           strict
               Documentation improved
    
           Tie::Hash
               Documentation improved
    
           Time::HiRes
           Unicode::Collate
           Unicode::Normalize
           UNIVERSAL
               Documentation improved
    
    Note that libnet is among them. It could therefore well be that the use of sockets and threads under Win32 is tickling a bug.

    So claiming before that it most definitely was in threads->detach is simply not true. At least not by itself.

    I have severe criticism on the implementation of threads in Perl. You will find some of it at Things you need to know before programming Perl ithreads. And I too have been frustrated at the seemingly random way of crashing that threaded Perl programs sometimes have. But in the end, there's a reason for it: by using threads, it just becomes an order of magnitude harder to trace and fix.

    When you're debugging threaded programs, you need to develop a brain that can run several processes simultaneously, which is pretty hard if you're used to just needing to follow a single process' logic. At every point in your code, you need to make sure that there can not be another thread intervening.

    Some (core) libraries may not be there yet. libnet may be one of them.

    One final note: since you seem to be using Activestate's Perl (I'm assuming here, since you said you downloaded a binary for Win32), it may be wise to also alert ActiveState of your problem as well: most of the Win32 Perl expertise is located at ActiveState, and interaction with p5p has recently become what I would call "suboptimal".

    One more final note: don't count on it being fixed for 5.8.3. Nicholas Clark has indicated that only stable bug fixes will be allowed in at this point. Without even an indication of the cause of the problem, I don't think a bug fix will be stable enough to be deemed ok by Nick. On the bright side: I wouldn't be surprised if the problem is in libnet, which would imply a version of CPAN (or ActiveState's ppm repository) could become available quickly after the problem has been found and fixed.

    Liz

      liz, please see my reply above to castaway, so it breaks without socket. (You did made me thought that it might be the combination of socket and thread caused problems, and that's why I intentionally came up with something really really simple.)

      But as I said in our conversation, I do appreciate your comment, and it has such a rich content.

      My thought sometime jumps too quick (as everyone knows me said), and I really like your style: calm and with lots of thought.

      Thanks again, for not only this reply, but also your reply to my first notes on .2 threads.

      most of the Win32 Perl expertise is located at ActiveState, and interaction with p5p has recently become what I would call "suboptimal".

      Thats an interesting comment that Id like to hear expanded upon if you feel like doing so....


      ---
      demerphq

        First they ignore you, then they laugh at you, then they fight you, then you win.
        -- Gandhi


Re: Perl 5.8.2 thread is worse - more findings from today
by Ninthwave (Chaplain) on Dec 31, 2003 at 09:17 UTC

    For me with ActiveState 5.8.2 on Windows 2000 the dropped scalars and memory leaks I was having with code I was working on is gone and my threads are running much better. I will look at the code you have but I guess some of the improvements I am having broke other things, but that has been the caveat with threads, ithreads particularly. This is an experiment and it may change dramatacially with each release. So is it a bug or is just the way ithreads are going. For me I am enjoying the change (at the moment).

    As a general question since ithreads are an experiment that is wanted to be included is there any area within the Perl community to feedback on this subsystem specifically???

    "No matter where you go, there you are." BB
      As a general question since ithreads are an experiment that is wanted to be included is there any area within the Perl community to feedback on this subsystem specifically???

      Check out http://lists.perl.org for the iThreads mailing list.

      Liz

Re: Perl 5.8.2 thread is worse - more findings from today
by liz (Monsignor) on Jan 01, 2004 at 11:59 UTC
    Bad news in the new year, I'm afraid. The bug is known, pretty old, and won't get fixed by 5.8.3. And I think I need to retract my comment about the communication between p5p and ActiveState being "suboptimal".

    Now, I could envision a temporary hack which would use Thread::Exit and that would steal threads->detach to do nothing except marking it as "detached", and stealing threads->new to join() all the threads marked as "detached", so that at least the memory would be freed of the finished threads. I could also put this into a seperate module, e.g. Thread::Detach, which would disable itself for future versions of Perl where this problem would fixed on Win32.

    Would that help?

    Liz

      It rather sounds like they should be using Thread Local Storage*, instead of Global Memory* or Heap memory*. The problem is, the design of perl's memory management doesn't make it easy to use multiple memory allocation strategies for different portions of the interpreter.

      *links won't work correctly unless your browser pretends to be (or is) IE or netscape.


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
      Hooray!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://317281]
Approved by bassplayer
Front-paged by Courage
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (8)
As of 2014-12-20 04:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (95 votes), past polls