Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Re^9: memory leaks with threads

by BrowserUk (Pope)
on Jul 09, 2007 at 23:33 UTC ( #625730=note: print w/replies, xml ) Need Help??


in reply to Re^8: memory leaks with threads
in thread memory leaks with threads

It's no great surprise that it's not the system apis that are at fault, but the perl source code on top of them.

I'll however wait a bit before submitting a bug report, maybe someone else knows a solution.

Okay, but if the last version of code I posted leaks that badly on 5.9.5, it's most unlikely that there is anything that can be done at Perl level (ie. in your code) to 'fix' the problem. You're going to have to wait at least until dave_the_m and co. at p5p track down and fix whatever is wrong. That's why I suggested the perlbug to ensure that a) it comes to their attention; b) they get all the relevant information they need.

Looking at the problem from a different angle. As you saw by comparing the speed and memory usage of pthreads at the C level, and ithreads at the perl level, ithreads are very heavy by comparison. That's easily explained by the benefits they give you--automatic memory management, explicitly shared, rather than implicitly shared variable spaces, and everything else that is Perl over C. And the work that is involved in achieving those--basically, starting a new interpreter and cloning the current environment each time you start a thread.

What all that is building up to, is that spawning threads for small amounts of work and then throwing them away--whilst very effective in C--is not the best way of tackling a threading problem using ithreads and perl. A better way is to use a pool of threads and re-use them.


Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

Replies are listed 'Best First'.
Re^10: memory leaks with threads
by misc (Pilgrim) on Jul 10, 2007 at 13:34 UTC
    I submitted a bugreport.

    I did some benchmarking,
    and calculated the time of running a function 300 times.

    The results are, again, most suprising to me.

    With 10 worker threads in a pool I get this:
    micha@laptop ~/prog/perl/test $ time ./threads_benchmark.pl real 0m36.666s user 1m11.872s sys 0m0.172s (resident memory 6.3MB)



    And, creating and destroying 300 threads, 10concurrent threads
    micha@laptop ~/prog/perl/test $ time ./threadpool.pl real 0m33.001s user 1m4.728s sys 0m0.024s (max res memory indefinite rising, 10MB)


    Ok, my threadpool implementation is far from beeing perfect, but I had guessed the differences in speed would be more obvious.

    Btw, I benchmarked also Thread::Pool

    Thread::Pool
    micha@laptop ~/prog/perl/test $ time ./thread_pool_benchmark.pl real 0m36.105s user 1m9.780s sys 0m0.048s 15M res memory




    My thread pool implementation (the first benchmark and code listed) seems to be stable and not too bad,
    I believe there are already some advantages to Thread::Pool.

    Is there a proposed way to publish such code, perhaps in Snippets ?

    Update: Quoted the scripts in readmore tags, as suggested, I'm also going to publish my ongoing work in my scratchpad
    Comments welcome
      Is there a proposed way to publish such code, perhaps in Snippets ?

      Personally, I think adding your code (all 3 versions) to your post above in separate <readmore><code>...</code></readmore> blocks would be good. It'd put the benchmark figures into context, and make the code available for searching for those that follow.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://625730]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (6)
As of 2018-08-21 21:12 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    Asked to put a square peg in a round hole, I would:









    Results (203 votes). Check out past polls.

    Notices?