|Welcome to the Monastery|
I was just wondering if there was any logic that went behind your suggestion of 100 threads for 8GB of memory if that was just a rough estimation?
A simple guestimation based upon my observation that on my system, each client requesting a dir /s c:\, required around 50/60 MB in order to accumulate all the output, wrap it up and forward it to the client. 100 * 60MB ~= 6GB leaving some headroom for other stuff. Also, remember that there is a fixed overhead for the OS, so 100 on 8GB might well translate to < 50 on 4GB.
I think that the real resource problem with your server/protocol is the need to accumulate all the output at the server prior to returning it to the client, forced on you in part by your use of backticks to execute the command.
If you used a piped-open and returned the output to the client line by line as you get it:
then your server memory usage would be cut to a 1/10th of its current requirements with (hopefully) pro-rata benefits to the number of concurrent clients you could handle. But I realise that would require a substantial re-working of both your server processing and the communications protocol.
The upside of the change would be that your server's concurrent client limits would be independent of commands they are running (and volumes of output they produce), as you would only cache a single line at the server. It would also allow your clients to start seeing the output from their interactions in much closer to real time. And potentially even interrupt that output if they've seen enough.
Also, transmitting the retrieved output line by line would have far less impact upon the network infrastructure than returning it in one huge chunk.
I also wonder if you have the possibility to try your tests on a real machine rather than a VM? I suspect that if you did, you would see far fewer of these kinds of "mysterious OS problems". That based on my own observations of weirdnesses with code running in VMs.
You might also consider upgrading the OS. WS-2003 predates most of the rise and rise of VMs, and I'm sure that the use of VMs has highlighted (and hopefully caused to be fixed) many dubious practices in the earlier kernels. WS-2010 might be more stable in that environment.
In a similar vein, I found far fewer problems running VMs under Vista than I did under XP. And more modern processors with the various levels of VT-x/AMD-V extensions are less prone to such mysteries than older ones.
Thanks again for all your help, I'll post again if I run into the same problem
You're welcome and good luck. (And it is always nice to get feedback:)
With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
In reply to Re^11: PANIC: underlying join failed threded tcp server