|Perl: the Markov chain saw|
Re: Use more threads.by renodino (Curate)
|on Feb 27, 2006 at 16:05 UTC||Need Help??|
renodino has done some testing with home built versions of Perl for Linux...
Clarification: I didn't build a new Perl on Linux. Using the stock Perl 5.8.6 in FC4, I ran some tests. Linux died at 289 threads. I also ran tests on Solaris 10 (which dies at ~1900 threads, and starts thrashing the swapper around 1300 threads), and OS X 10.3.9, which dies around 450 threads.
Perhaps as importantly, I found a link that sheds a bit more light on the subject.
My current approach (which I hope to build/test today) is to add a couple new APIs to threads: set_stack_size() and get_stack_size(). The added code is pretty simple, though it may not be applicable to the root thread (the various editbin/setrlimit/ulimit solutions may address that issue).
Its important to point out that this issue isn't just about using more threads (tho thats my personal requirement); given the huge default stack size on Win32 and Linux, one of the biggest complaints of threaded Perl apps - its voracious memory appetite - may be addressed by just trimming the stack reserve down to a reasonable/minimal number.
After adding the set/get_stack_size() methods and applying the associated changes to the CreateThread()/pthread_attr_setstacksize() calls, and then calling set_stack_size(65536), I can crank out 1200 threads on Win32 (tho theres definitely some swapping kicking in at around 900 threads).
Likewise, on Linux FC4, I can get 1000 threads on a fairly small machine (an old 1GHz laptop w/512 meg), tho it starts thrashing at around 1000 threads. And the vsz report from ps shows a vast reduction in memory usage. (Since I can't get more than 120 threads using the original threads on Win32, I can't really make a useful memory usage comparison)
Note that in both cases, I'm using the stock perl 5.8.6 wo/ any ulimit'ing or editbin'ing.
I'm going to try it on OS X and Solaris and see what shakes out.
FWIW: my method for doing this was to copy the threads and threads::shared source directories into their own, and rename everything to "morethreads" package root. The module tests don't seem to pass w/ flying colors, but it may be related to using the unofficial threads::shared 0.95 against perl 5.8.6.
After testing on OS X 10.3.9 and Solaris 10, they both seem a bit less sensitive to the stack setting. Both reported ulimit -s == 8192 (ie, 8Meg).
When I ran a comparison test on OS X between stock threads, and my hacked morethreads, the overal performance was about the same, tho ps -o vsz reported about half as much memory being used when I set_stack_size(65536). So I'm assuming something in either the perl build, or the OS is throttling the per-thread stack size.
On Solaris, the test showed an even closer vsz between stock and hacked threads. Stock was always about 15-20 megs higher than hacked, so I'm assuming theres a build or OS limit there as well.
Following up on my Linux tests, ulimit -s reported 10240. The vsz differences were dramatic: at 200 threads, the stock version reported nearly 2Gig, while the hacked version reported around 125Meg.