Re^6: Multithreaded Server Crashes under heavy load

[rmahin]

Good luck with your account!

Ok collected the information again. Sorry, the other 300 threads were from another a perl process running on the machine, but was just not visible on my remote desktop session, should have figured they would show up in the log though, my mistake. Ran this one on another machine, with no other users accessing it, and no other perl programs running. The 38 hours you saw was also for the process. Nonetheless, the log I posted only covered a 3 minute period because thats just how long it takes it to crash under the stress testing I'm doing. We had seen it happen occasionally under our regular use, so to flush out the problems I created some tests to get it to happen quicker, as the time it took for it to happen seemed entirely random, sometimes a day or two. This log also only lasts a few minutes, and I did it over 10 second intervals. Some commands it executes will be as simple as making a directory, and take almost no time at all to process, and others are longer.

https://dl.dropbox.com/u/19686501/RXD_000004.csv

Also, I just cant really see any reason why a thread would not be exiting properly. When I run the program in debug mode, it seems to crash almost instantly which would lead me to believe could have something to do with printing to the screen, but have tried redirecting all output to a file with no change. But what is printed with the debug messages shows that all commands that the server actually receives and executes run, and close successfully. Here's the output of all the commands before it crashed https://dl.dropbox.com/u/19686501/debug.log. It shows the "DEBUG -- (30) Closing connection" printed out by

    debug("Closing connection");
    shutdown($client, 2);
    close $client;
    threads->exit;

at the end of thread subroutine. So, yeah I'm really at a loss.

Again, thank you for your help