Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Thread creation failed: pthread_create returned 11

by kamrul (Acolyte)
on Aug 10, 2015 at 23:49 UTC ( #1138090=perlquestion: print w/replies, xml ) Need Help??

kamrul has asked for the wisdom of the Perl Monks concerning the following question:

My code is like below:
my $tcount :shared = 0; while( $message = $msgQueue->poll()){ if($tcount < 4){ Logger::log(3, "Number of active threads : ".$tcount." pro +cessing incomming message"); threads->create(processData,$message,createMsgId()); } else { Logger::log(2, "Maximum number of threads reached. Waiting +"); sleep 1 until $tcount < 4); threads->create(processData,$message,createMsgId()); } } sub processData { $tcount++; # do something $tcount--; threads->exit(); }
As per my understanding I am not creating more than 4 threads at a time. However, after running this script for a while I get: Thread creation failed: pthread_create returned 11 What wrong am I doing?

Replies are listed 'Best First'.
Re: Thread creation failed: pthread_create returned 11 (races, leaks)
by tye (Sage) on Aug 11, 2015 at 05:11 UTC
    I am not creating more than 4 threads at a time.

    Actually, you could create unbounded many threads between the call to threads->create() and when that thread finally gets created and runs far enough to execute $tcount++.

    You could certainly reduce that potential problem by moving the duplicate threads->create() calls out of the if/else blocks so you have just one copy of it below that and do the $tcount++ there instead of inside of processData(). You could still potentially accumulate unbounded many threads whose execution lies somewhere between the $tcount-- and the actual ending of the thread.

    You also don't appear to ever join() nor detach() any of these threads. That surely builds up a growing stash of information about what each thread "returned", much like a memory leak.

    These types of problems are part of why the usual pattern is to just create 4 threads and then farm out items to them to execute (usually using a thread queue).

    - tye        

      Based on what you said, it seems those threads take a bit of time to get created. In that case, will it be more efficient if I keep 4 running threads and pass the messages to the threads using a queue; rather than creating new threads every time ?
Re: Thread creation failed: pthread_create returned 11 (RTFM)
by tye (Sage) on Aug 11, 2015 at 00:47 UTC

    Go peruse man pthread_create but for your particular operating system.

    Since I get "Resource temporarily unavailable" for $! = 11 on two rather different systems, there is a good chance that 11 means that on your system as well. And your version of the man page should explain much more explicitly what are the causes for that error reason.

    - tye        

Re: Thread creation failed: pthread_create returned 11
by BrowserUk (Pope) on Aug 11, 2015 at 00:18 UTC

    If you'd care to provide a runnable program -- that uses strict & warnings and compiles clean -- I'd be willing to take a look at the problem; but I'm not enamored to do so whilst your code contains errors like these:

    C:\test>junk77 Global symbol "$message" requires explicit package name at C:\test\jun +k77.pl line 9. Global symbol "$msgQueue" requires explicit package name at C:\test\ju +nk77.pl line 9. Global symbol "$message" requires explicit package name at C:\test\jun +k77.pl line 12. syntax error at C:\test\junk77.pl line 15, near "4)" Global symbol "$message" requires explicit package name at C:\test\jun +k77.pl line 16. Bareword "processData" not allowed while "strict subs" in use at C:\te +st\junk77.pl line 12. Bareword "processData" not allowed while "strict subs" in use at C:\te +st\junk77.pl line 16. Execution of C:\test\junk77.pl aborted due to compilation errors.

    Which when I attempt to fix them in the obvious way results in this error:

    C:\test>junk77 Can't locate object method "poll" via package "Thread::Queue" at C:\te +st\junk77.pl line 12.

    On the basis on what you've posted, I cannot see how you could ever get the error message you cite.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
    In the absence of evidence, opinion is indistinguishable from prejudice.
    I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!
      This should reproduce the problem:
      use threads; use threads::shared; my $tcount :shared = 0; while(1){ if($tcount < 4){ print("Number of active threads : ".$tcount." processing incom +ming message\n"); threads->create(processData); } else { print("Maximum number of threads reached. Waiting \n"); sleep 1 until $tcount < 4; threads->create(processData); } } sub processData { $tcount++; #sleep(2); print "OK \n"; $tcount--; threads->exit(); }

        You are simply exhausting your memory because you are never joining your threads.

        The easiest fix is to just detach your threads so they get cleaned up automatically. (Also, there is no point in calling threads->exit; better to just fall off the end.

        So whilst your example above chews up 8GB of memory in around a minute or so, this slightly modified version runs with no signs of memory growth at all:

        #! perl -sw #use strict; use threads; use threads::shared; my $tcount :shared = 0; while(1){ if($tcount < 4){ print("Number of active threads : ".$tcount." processing incom +ming message\n"); threads->create(processData)->detach; } else { print("Maximum number of threads reached. Waiting \n"); sleep 1 until $tcount < 4; threads->create(processData)->detach; } } sub processData { $tcount++; #sleep(2); print "OK \n"; $tcount--; }

        There are several other bad things and weirdnesses in that short snippet -- most of which would be cured by making it strict/warnings safe; but I guess that request fell on deaf ears -- but it will cure your headline problem.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority". I knew I was on the right track :)
        In the absence of evidence, opinion is indistinguishable from prejudice.
        I'm with torvalds on this Agile (and TDD) debunked I told'em LLVM was the way to go. But did they listen!

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1138090]
Approved by Old_Gray_Bear
Front-paged by BrowserUk
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (8)
As of 2020-04-03 18:13 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    The most amusing oxymoron is:
















    Results (31 votes). Check out past polls.

    Notices?