Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

Re^8: memory leaks with threads

by misc (Pilgrim)
on Jul 09, 2007 at 20:07 UTC ( #625705=note: print w/ replies, xml ) Need Help??


in reply to Re^7: memory leaks with threads
in thread memory leaks with threads

I'm running Linux, 2.6.18, 32bit on a turion x2 tl-60.

I just ran a similar program written in c.
It doesn't grow (stays exactly at a res memory of 944B), also after 15 million threads created and destroyed.
So I can at least say it's not related to my pthreads library.

I'll however wait a bit before submitting a bug report, maybe someone else knows a solution.

#include <iostream> #include <cstdlib> using namespace std; #include <pthread.h> #include <unistd.h> #include <sys/select.h> #include <sys/time.h> using namespace std; static int threadscount = 0; pthread_mutex_t threadscount_mutex = PTHREAD_MUTEX_INITIALIZER; void* thread( void* data ){ pthread_mutex_lock( &threadscount_mutex ); threadscount--; //cout << threadscount << endl; pthread_mutex_unlock( &threadscount_mutex ); pthread_exit(0); } int main(int argc, char *argv[]){ int count = 0; int a; struct timeval timeout; while (1){ for ( int a = 1; a <= 10; a++ ){ pthread_mutex_lock( &threadscount_mutex ); threadscount++; pthread_mutex_unlock( &threadscount_mutex ); pthread_t p_thread; /* thread's structu +re */ pthread_create(&p_thread, 0, thread, (void*)&a +); pthread_detach( p_thread ); count ++; } cout << "count: " << count << endl; int c; do { timeout.tv_usec = 100; timeout.tv_sec = 0; select( 0, 0, 0, 0, &timeout ); pthread_mutex_lock( &threadscount_mutex ); c = threadscount; pthread_mutex_unlock( &threadscount_mutex ); } while ( c > 0); } }


Comment on Re^8: memory leaks with threads
Download Code
Re^9: memory leaks with threads
by BrowserUk (Pope) on Jul 09, 2007 at 23:33 UTC

    It's no great surprise that it's not the system apis that are at fault, but the perl source code on top of them.

    I'll however wait a bit before submitting a bug report, maybe someone else knows a solution.

    Okay, but if the last version of code I posted leaks that badly on 5.9.5, it's most unlikely that there is anything that can be done at Perl level (ie. in your code) to 'fix' the problem. You're going to have to wait at least until dave_the_m and co. at p5p track down and fix whatever is wrong. That's why I suggested the perlbug to ensure that a) it comes to their attention; b) they get all the relevant information they need.

    Looking at the problem from a different angle. As you saw by comparing the speed and memory usage of pthreads at the C level, and ithreads at the perl level, ithreads are very heavy by comparison. That's easily explained by the benefits they give you--automatic memory management, explicitly shared, rather than implicitly shared variable spaces, and everything else that is Perl over C. And the work that is involved in achieving those--basically, starting a new interpreter and cloning the current environment each time you start a thread.

    What all that is building up to, is that spawning threads for small amounts of work and then throwing them away--whilst very effective in C--is not the best way of tackling a threading problem using ithreads and perl. A better way is to use a pool of threads and re-use them.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      I submitted a bugreport.

      I did some benchmarking,
      and calculated the time of running a function 300 times.

      The results are, again, most suprising to me.

      With 10 worker threads in a pool I get this:
      micha@laptop ~/prog/perl/test $ time ./threads_benchmark.pl real 0m36.666s user 1m11.872s sys 0m0.172s (resident memory 6.3MB)



      And, creating and destroying 300 threads, 10concurrent threads
      micha@laptop ~/prog/perl/test $ time ./threadpool.pl real 0m33.001s user 1m4.728s sys 0m0.024s (max res memory indefinite rising, 10MB)


      Ok, my threadpool implementation is far from beeing perfect, but I had guessed the differences in speed would be more obvious.

      Btw, I benchmarked also Thread::Pool

      Thread::Pool
      micha@laptop ~/prog/perl/test $ time ./thread_pool_benchmark.pl real 0m36.105s user 1m9.780s sys 0m0.048s 15M res memory




      My thread pool implementation (the first benchmark and code listed) seems to be stable and not too bad,
      I believe there are already some advantages to Thread::Pool.

      Is there a proposed way to publish such code, perhaps in Snippets ?

      Update: Quoted the scripts in readmore tags, as suggested, I'm also going to publish my ongoing work in my scratchpad
      Comments welcome
        Is there a proposed way to publish such code, perhaps in Snippets ?

        Personally, I think adding your code (all 3 versions) to your post above in separate <readmore><code>...</code></readmore> blocks would be good. It'd put the benchmark figures into context, and make the code available for searching for those that follow.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://625705]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (10)
As of 2014-12-19 14:33 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (84 votes), past polls