Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
renodino has done some testing with home built versions of Perl for Linux...

Clarification: I didn't build a new Perl on Linux. Using the stock Perl 5.8.6 in FC4, I ran some tests. Linux died at 289 threads. I also ran tests on Solaris 10 (which dies at ~1900 threads, and starts thrashing the swapper around 1300 threads), and OS X 10.3.9, which dies around 450 threads.

Perhaps as importantly, I found a link that sheds a bit more light on the subject.

My current approach (which I hope to build/test today) is to add a couple new APIs to threads: set_stack_size() and get_stack_size(). The added code is pretty simple, though it may not be applicable to the root thread (the various editbin/setrlimit/ulimit solutions may address that issue).

Its important to point out that this issue isn't just about using more threads (tho thats my personal requirement); given the huge default stack size on Win32 and Linux, one of the biggest complaints of threaded Perl apps - its voracious memory appetite - may be addressed by just trimming the stack reserve down to a reasonable/minimal number.

Update:

After adding the set/get_stack_size() methods and applying the associated changes to the CreateThread()/pthread_attr_setstacksize() calls, and then calling set_stack_size(65536), I can crank out 1200 threads on Win32 (tho theres definitely some swapping kicking in at around 900 threads).

Likewise, on Linux FC4, I can get 1000 threads on a fairly small machine (an old 1GHz laptop w/512 meg), tho it starts thrashing at around 1000 threads. And the vsz report from ps shows a vast reduction in memory usage. (Since I can't get more than 120 threads using the original threads on Win32, I can't really make a useful memory usage comparison)

Note that in both cases, I'm using the stock perl 5.8.6 wo/ any ulimit'ing or editbin'ing.

I'm going to try it on OS X and Solaris and see what shakes out.

FWIW: my method for doing this was to copy the threads and threads::shared source directories into their own, and rename everything to "morethreads" package root. The module tests don't seem to pass w/ flying colors, but it may be related to using the unofficial threads::shared 0.95 against perl 5.8.6.

Update 2:

After testing on OS X 10.3.9 and Solaris 10, they both seem a bit less sensitive to the stack setting. Both reported ulimit -s == 8192 (ie, 8Meg).

When I ran a comparison test on OS X between stock threads, and my hacked morethreads, the overal performance was about the same, tho ps -o vsz reported about half as much memory being used when I set_stack_size(65536). So I'm assuming something in either the perl build, or the OS is throttling the per-thread stack size.

On Solaris, the test showed an even closer vsz between stock and hacked threads. Stock was always about 15-20 megs higher than hacked, so I'm assuming theres a build or OS limit there as well.

Following up on my Linux tests, ulimit -s reported 10240. The vsz differences were dramatic: at 200 threads, the stock version reported nearly 2Gig, while the hacked version reported around 125Meg.


In reply to Re: Use more threads. by renodino
in thread Use more threads. by BrowserUk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others drinking their drinks and smoking their pipes about the Monastery: (14)
    As of 2014-10-24 10:38 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      For retirement, I am banking on:










      Results (131 votes), past polls