Beefy Boxes and Bandwidth Generously Provided by pair Networks
more useful options
 
PerlMonks  

Re^2: Perl detached threads, loops & my?

by expresspotato (Beadle)
on Jun 09, 2010 at 00:00 UTC ( #843763=note: print w/ replies, xml ) Need Help??


in reply to Re: Perl detached threads, loops & my?
in thread Perl detached threads, loops & my?

Hello, I am using linux Fedora 12. The following script will consume all available memory in a matter of seconds.

use threads; while ($c < 100000){ @thr_ll[$c]=threads->new(\&sys_call,$c); @thr_ll[$c]->detach(); $c++; } sub sys_call(){ print "Called: $_[0]\n"; }
I know the above statement is the WRONG way of using threads but its the only way I know of ensuring a fresh thread each time. It will use all system memory in a matter of seconds. With the following code the program will eventually crash after being run for extended periods (HOURS) of time with other running threads. No error message is observed.
use threads; while ($c < 100000){ my $thr=threads->new(\&sys_call,$c); $thr->detach(); $c++; } sub sys_call(){ print "Called: $_[0]\n"; }


Comment on Re^2: Perl detached threads, loops & my?
Select or Download Code
Re^3: Perl detached threads, loops & my?
by BrowserUk (Pope) on Jun 09, 2010 at 00:19 UTC

    I'm sorry, but you have an unconstrained flat out loop creating hundreds of thousands of threads ... and you wonder why it runs out of memory?

    Why are you storing the thread handles when you're detaching them?

    Did you not see me ask which version of Perl you are using?

    ... but its the only way I know of ensuring a fresh thread each time.

    What on earth do you mean by a "fresh thread"? And why are you so certain that you need them?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
Re^3: Perl detached threads, loops & my?
by rjt (Deacon) on Jun 09, 2010 at 00:27 UTC

    Thank you for the example. However, I still have doubts. Please carefully consider this reply, my other reply, and BrowserUk's comments.

    Red flag for me: Your second script, you claim takes "HOURS" to run. On the below mentioned old system, that script completes for me in minutes. Process memory usage never went above 50MB.

    $ time perl script.pl Called: 0 Called: 1 : Called: 99998 Called: 99999 real 12m23.351s user 10m50.800s sys 0m23.580s

    You're trying to create 100000 threads. You invite a race condition where your script may hit a wall very quickly if your threads do anything non-trivial. You would expect to see:

    Called: 367 Called: 368 Thread creation failed: pthread_create returned 11 at script line 4. Can't call method "detach" on an undefined value at script line 5.

    Your first example will "leak" memory in the sense that the @thr_ll array will grow to 100000, even after threads exit. On most systems this wouldn't be a big deal if your real application caps at 100000 as well, but I still wouldn't recommend it.

    I ran both of your scripts on my oldest machine, and neither one had a deleterious effect on Perl 5.10.1 on Ubuntu, running on an 8 year old P4-1.6GHz machine with 512MB. Both ran to completion.

Re^3: Perl detached threads, loops & my?
by llancet (Friar) on Jun 09, 2010 at 01:13 UTC
    I'm using a powerful computer, with 2 6-core cpus, and 8G memory:
    $ uname -a Linux localhost.localdomain 2.6.33.5-112.fc13.x86_64 #1 SMP Thu May 27 + 02:28:31 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux
    with a somehow new version perl and threads:
    $ perl -v This is perl, v5.10.1 (*) built for x86_64-linux-thread-multi Copyright 1987-2009, Larry Wall Perl may be copied only under the terms of either the Artistic License + or the GNU General Public License, which may be found in the Perl 5 source ki +t. Complete documentation for Perl, including FAQ lists, should be found +on this system using "man perl" or "perldoc perl". If you have access to + the Internet, point your browser at http://www.perl.org/, the Perl Home Pa +ge. $ perl -Mthreads -e 'print $threads::VERSION,"\n"' 1.77
    Actually, if I do something like your second example, it would core dump immediately:
    $ perl -e 'use threads; for(1..100000) { my $t = threads->new(\&body,$ +_); $t->detach; } sub body {print "Called: @_\n"}'
    Called: 1 Called: 2 Called: 3 Called: 4 Called: 5 Called: 6 Called: 7 Called: 8 Called: 9 Called: 10 Called: 11 Called: 12 Called: 13 Called: 14 Called: 15 Called: 16 Called: 17 Called: 18 Called: 19 Called: 20 Called: 21 Called: 22 Called: 23 Called: 24 Called: 25 Called: 26 Called: 27 Called: 28 Called: 29 Called: 30 Called: 31 Called: 32 Called: 33 Called: 34 Called: 35 Called: 36 Called: 37 Called: 38 Called: 39 Called: 40 Called: 41 Called: 42 Called: 43 Called: 44 Called: 45 Called: 46 Called: 47 Called: 48 Called: 49 Called: 50 Called: 51 Called: 52 Called: 53 Called: 54 Called: 55 Called: 56 Called: 57 Called: 58 Called: 59 Called: 60 Called: 61 Called: 62 Called: 63 Called: 64 Called: 65 Called: 66 Called: 67 Called: 68 Called: 69 Called: 70 Called: 71 Called: 72 Called: 73 Called: 74 Called: 75 Called: 76 Called: 77 Called: 78 Called: 79 Called: 80 Called: 81 Called: 82 Called: 83 Called: 84 Called: 85 Called: 86 Called: 87 Called: 88 Called: 89 Called: 90 Called: 91 Called: 92 *** glibc detected *** perl: invalid fastbin entry (free): 0x000000000 +0789010 *** ======= Backtrace: ========= /lib64/libc.so.6[0x3b8fc75746] /usr/lib64/perl5/CORE/libperl.so[0x3b8f89de8f] /usr/lib64/perl5/CORE/libperl.so(Perl_hv_free_ent+0x98)[0x3b8f89e398] /usr/lib64/perl5/CORE/libperl.so[0x3b8f89f861] /usr/lib64/perl5/CORE/libperl.so(Perl_hv_undef+0x68)[0x3b8f8a1648] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x5c3)[0x3b8f8b8e23] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112] /usr/lib64/perl5/CORE/libperl.so(Perl_gp_free+0x19d)[0x3b8f84f13d] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x70d)[0x3b8f8b8f6d] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112] /usr/lib64/perl5/CORE/libperl.so(Perl_hv_free_ent+0x42)[0x3b8f89e342] /usr/lib64/perl5/CORE/libperl.so[0x3b8f89f861] /usr/lib64/perl5/CORE/libperl.so(Perl_hv_undef+0x68)[0x3b8f8a1648] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x5c3)[0x3b8f8b8e23] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112] /usr/lib64/perl5/CORE/libperl.so(Perl_gp_free+0x19d)[0x3b8f84f13d] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_clear+0x70d)[0x3b8f8b8f6d] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_free2+0x52)[0x3b8f8b9112] /usr/lib64/perl5/CORE/libperl.so[0x3b8f8ae131] /usr/lib64/perl5/CORE/libperl.so(Perl_sv_clean_all+0x1b)[0x3b8f8ae1db] /usr/lib64/perl5/CORE/libperl.so(perl_destruct+0xb79)[0x3b8f84e269] /usr/lib64/perl5/auto/threads/threads.so(+0x5496)[0x7f7b9d360496] /usr/lib64/perl5/auto/threads/threads.so(+0x5682)[0x7f7b9d360682] /usr/lib64/perl5/auto/threads/threads.so(+0x6849)[0x7f7b9d361849] /lib64/libpthread.so.0[0x3b90407761] /lib64/libc.so.6(clone+0x6d)[0x3b8fce150d] ======= Memory map: ======== 00400000-00401000 r-xp 00000000 08:21 3678850 + /usr/bin/perl 00601000-00603000 rw-p 00001000 08:21 3678850 + /usr/bin/perl 00624000-00e46000 rw-p 00000000 00:00 0 + [heap] 3b8f400000-3b8f41e000 r-xp 00000000 08:21 3017770 + /lib64/ld-2.12.so 3b8f61e000-3b8f61f000 r--p 0001e000 08:21 3017770 + /lib64/ld-2.12.so 3b8f61f000-3b8f620000 rw-p 0001f000 08:21 3017770 + /lib64/ld-2.12.so 3b8f620000-3b8f621000 rw-p 00000000 00:00 0 3b8f800000-3b8f962000 r-xp 00000000 08:21 3727883 + /usr/lib64/perl5/CORE/libperl.so 3b8f962000-3b8fb62000 ---p 00162000 08:21 3727883 + /usr/lib64/perl5/CORE/libperl.so 3b8fb62000-3b8fb6b000 rw-p 00162000 08:21 3727883 + /usr/lib64/perl5/CORE/libperl.so 3b8fc00000-3b8fd75000 r-xp 00000000 08:21 3017771 + /lib64/libc-2.12.so 3b8fd75000-3b8ff75000 ---p 00175000 08:21 3017771 + /lib64/libc-2.12.so 3b8ff75000-3b8ff79000 r--p 00175000 08:21 3017771 + /lib64/libc-2.12.so 3b8ff79000-3b8ff7a000 rw-p 00179000 08:21 3017771 + /lib64/libc-2.12.so 3b8ff7a000-3b8ff7f000 rw-p 00000000 00:00 0 3b90000000-3b90002000 r-xp 00000000 08:21 3017773 + /lib64/libdl-2.12.so 3b90002000-3b90202000 ---p 00002000 08:21 3017773 + /lib64/libdl-2.12.so 3b90202000-3b90203000 r--p 00002000 08:21 3017773 + /lib64/libdl-2.12.so 3b90203000-3b90204000 rw-p 00003000 08:21 3017773 + /lib64/libdl-2.12.so 3b90400000-3b90417000 r-xp 00000000 08:21 3017779 + /lib64/libpthread-2.12.so 3b90417000-3b90616000 ---p 00017000 08:21 3017779 + /lib64/libpthread-2.12.so 3b90616000-3b90617000 r--p 00016000 08:21 3017779 + /lib64/libpthread-2.12.so 3b90617000-3b90618000 rw-p 00017000 08:21 3017779 Ca +lled: 93 Aborted (core dumped)

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://843763]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2014-09-15 04:31 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (145 votes), past polls