in reply to Memory management
Perl Threads are a joke. The perl engine NEVER releases RAM to the OS. Why? Who knows. One explanation I've heard is RAM fragmentation or that perl never sees a totally free block of RAM again to hand back to the OS. The other is that perl does its own memory management, and it mallocs huge chunks of ram from the OS, and will only "free" them back to its internal memory allocator. The internal memory allocater will NEVER free them back to the OS. Its the problem of perl being the only implementation of Perl.
Ithreads are a joke. They receive little attention from perl developers since no "real" Perl user cares about threads since "all" "real" Perl users use Perl for CGI under POSIX which includes COW fork support and daemonization which remove the need for threads support in perl in perl developers minds/priorities. Perl ithreads is a 1 or 2 man job according to the readme, just 2 developers listed. Another thing that shows that perl developers only care about use perl for CGI and not for the desktop is that apache modperl compiles Perl to bytecode and saves the bytecode for later execution. The desktop version of this were B::C and Byteloader modules, which were removed in 2007 from the CORE since they weren't maintained and development stopped on them for a long time. Perl on the desktop and especially Windows receives very little support overall. From what I've seen, Activestate used to spend alot of money on perl on win32 in the late 1990s/early 2000s, but has given up on investing in perl since then. Activestate used to contribute a huge amount to the Perl world. Win32 patches, PerlNET, PerlScript, Perl in IIS, and countless other things.
Regarding your threads memory consumption problem, [rt://44493] talks about it. So does the POD for threads. Note the author of ithreads basically said he doesn't care about the problem/mem leak and its your fault and he isn't going to fix it. Fits the attitude of poor/no support of Perl on the desktop.
The problem with ithreads is that ithreads doesn't share 1 perl interpretor/op tree executor between multiple threads. Each perl ithread gets its own perl engine copy of x86 assembly in RAM and basically is a new perl process embedded in a new thread created in XS. A better design, in my amateur opinion would be 1 static perl interpreter, with each thread having its own stack/context and optree, but that won't happen since perl developers are perfectly happy with COW fork on POSIX and won't change perl to work on Windows.
Cygwin fork doesn't COW either. I'm not sure about Interix fork. Interix is the damn closest you can get to POSIX on windows. Interix is closed source, and its developers have access to the Windows source code and it uses the Native NT API that Cygwin and not other FOSS POSIX toolkits get to use. Try using perl on Interix and then doing fork and see how your RAM usage is compared to ithreads. Under the hood NT series is Unix or has all provisions to be Unix remember. The NT Kernel was made by a bunch of layed off DEC Unix engineers. The NT Kernel API is undocumented and unsupported by Microsoft, on purpose probably. Win32 API isn't Unix obviously.
In a normal programing language, there would be multiple Perl interpreters, and a Perl standard (although a fixed standard just wouldn't ever work with Perl ;)). Look at Javascript. Multiple engines. Multiple toolkits. Multiple enviroments. Look at Javascript, HTML, all Adobe products, Openoffice, JScript, J# (RIP), QTScript, all use Javascript, but are totally different things. Its sad that Perl is 23 years old, yet there is no IronPerl for .NET or any way to compile to x86 or bytecode on the desktop or anything but perl to run Perl. Perl 6 is a big part of the problem. Aslong as there is Perl 6, Perl and perl are a dead end. Its sad in 23 years Perl has had 2 forks (Ruby and PHP) because of developers being fed up with it for various reasons (someone elaborate if they have a plausible explanation).
perl also has no asynchronous File I/O on windows, well, it does, but its basically is undocumented in Win32API::File and has some cosmetic bugs. I've gotten it to work perfectly with workarounds for the cosmetic bugs. Putting my notes for doing async file I/O on Win32 on the internet is on my TODO list.
Your fix as said in ithread's POD is to reuse threads. Or just launch multiple perl.exes. What about using select() to handle multiple connections with a non-threaded perl? It supposedly work on win32.
In an ideal world, threads are NEVER needed in any algorithm, unless you want to use multiple processors. Before using the permabroken ithreads, ask yourself do you really need ithreads in the first place?
I heard there is an opinion that ithreads can never be used in a "production environment", and they exist solely for testing or developing POSIX Perl scripts on Windows, later to be transferred to a POSIX server for the "production environment".
Also what other modules are you using other than ithreads in your server Perl script? If LWP is one of them, thats your problem.
The code above shows that perl does release ram back to the OS according to task manager. It went to 200 MB of RAM in task manager on the 1st stop, after the 2nd stop it was at 13MB RAM. So in single threaded mode, perl will release RAM back to the OS on Windows.$t = "perlperlperlperlperlperlperlperlperlperlperlperlPP" x 20000; #1 + MB for (0..100) {$t3 .= $t;} system('pause'); undef($t3); undef($t); system('pause')
Ithreads are a joke. They receive little attention from perl developers since no "real" Perl user cares about threads since "all" "real" Perl users use Perl for CGI under POSIX which includes COW fork support and daemonization which remove the need for threads support in perl in perl developers minds/priorities. Perl ithreads is a 1 or 2 man job according to the readme, just 2 developers listed. Another thing that shows that perl developers only care about use perl for CGI and not for the desktop is that apache modperl compiles Perl to bytecode and saves the bytecode for later execution. The desktop version of this were B::C and Byteloader modules, which were removed in 2007 from the CORE since they weren't maintained and development stopped on them for a long time. Perl on the desktop and especially Windows receives very little support overall. From what I've seen, Activestate used to spend alot of money on perl on win32 in the late 1990s/early 2000s, but has given up on investing in perl since then. Activestate used to contribute a huge amount to the Perl world. Win32 patches, PerlNET, PerlScript, Perl in IIS, and countless other things.
Regarding your threads memory consumption problem, [rt://44493] talks about it. So does the POD for threads. Note the author of ithreads basically said he doesn't care about the problem/mem leak and its your fault and he isn't going to fix it. Fits the attitude of poor/no support of Perl on the desktop.
The problem with ithreads is that ithreads doesn't share 1 perl interpretor/op tree executor between multiple threads. Each perl ithread gets its own perl engine copy of x86 assembly in RAM and basically is a new perl process embedded in a new thread created in XS. A better design, in my amateur opinion would be 1 static perl interpreter, with each thread having its own stack/context and optree, but that won't happen since perl developers are perfectly happy with COW fork on POSIX and won't change perl to work on Windows.
Cygwin fork doesn't COW either. I'm not sure about Interix fork. Interix is the damn closest you can get to POSIX on windows. Interix is closed source, and its developers have access to the Windows source code and it uses the Native NT API that Cygwin and not other FOSS POSIX toolkits get to use. Try using perl on Interix and then doing fork and see how your RAM usage is compared to ithreads. Under the hood NT series is Unix or has all provisions to be Unix remember. The NT Kernel was made by a bunch of layed off DEC Unix engineers. The NT Kernel API is undocumented and unsupported by Microsoft, on purpose probably. Win32 API isn't Unix obviously.
In a normal programing language, there would be multiple Perl interpreters, and a Perl standard (although a fixed standard just wouldn't ever work with Perl ;)). Look at Javascript. Multiple engines. Multiple toolkits. Multiple enviroments. Look at Javascript, HTML, all Adobe products, Openoffice, JScript, J# (RIP), QTScript, all use Javascript, but are totally different things. Its sad that Perl is 23 years old, yet there is no IronPerl for .NET or any way to compile to x86 or bytecode on the desktop or anything but perl to run Perl. Perl 6 is a big part of the problem. Aslong as there is Perl 6, Perl and perl are a dead end. Its sad in 23 years Perl has had 2 forks (Ruby and PHP) because of developers being fed up with it for various reasons (someone elaborate if they have a plausible explanation).
perl also has no asynchronous File I/O on windows, well, it does, but its basically is undocumented in Win32API::File and has some cosmetic bugs. I've gotten it to work perfectly with workarounds for the cosmetic bugs. Putting my notes for doing async file I/O on Win32 on the internet is on my TODO list.
Your fix as said in ithread's POD is to reuse threads. Or just launch multiple perl.exes. What about using select() to handle multiple connections with a non-threaded perl? It supposedly work on win32.
In an ideal world, threads are NEVER needed in any algorithm, unless you want to use multiple processors. Before using the permabroken ithreads, ask yourself do you really need ithreads in the first place?
I heard there is an opinion that ithreads can never be used in a "production environment", and they exist solely for testing or developing POSIX Perl scripts on Windows, later to be transferred to a POSIX server for the "production environment".
Also what other modules are you using other than ithreads in your server Perl script? If LWP is one of them, thats your problem.
In Section
Seekers of Perl Wisdom