Beefy Boxes and Bandwidth Generously Provided by pair Networks vroom
Perl: the Markov chain saw
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
I was just wondering if there was any logic that went behind your suggestion of 100 threads for 8GB of memory if that was just a rough estimation?

A simple guestimation based upon my observation that on my system, each client requesting a dir /s c:\, required around 50/60 MB in order to accumulate all the output, wrap it up and forward it to the client. 100 * 60MB ~= 6GB leaving some headroom for other stuff. Also, remember that there is a fixed overhead for the OS, so 100 on 8GB might well translate to < 50 on 4GB.

I think that the real resource problem with your server/protocol is the need to accumulate all the output at the server prior to returning it to the client, forced on you in part by your use of backticks to execute the command.

If you used a piped-open and returned the output to the client line by line as you get it:

# $resp = `$rxdArgs 2>>&1`; my $pid = open my $PIPE, '-|' qq[ $rxdArgs 2>>&1 ] or + die $!; while( <$PIPE> ) { returnOutputToClient( $_ ); }

then your server memory usage would be cut to a 1/10th of its current requirements with (hopefully) pro-rata benefits to the number of concurrent clients you could handle. But I realise that would require a substantial re-working of both your server processing and the communications protocol.

The upside of the change would be that your server's concurrent client limits would be independent of commands they are running (and volumes of output they produce), as you would only cache a single line at the server. It would also allow your clients to start seeing the output from their interactions in much closer to real time. And potentially even interrupt that output if they've seen enough.

Also, transmitting the retrieved output line by line would have far less impact upon the network infrastructure than returning it in one huge chunk.

I also wonder if you have the possibility to try your tests on a real machine rather than a VM? I suspect that if you did, you would see far fewer of these kinds of "mysterious OS problems". That based on my own observations of weirdnesses with code running in VMs.

You might also consider upgrading the OS. WS-2003 predates most of the rise and rise of VMs, and I'm sure that the use of VMs has highlighted (and hopefully caused to be fixed) many dubious practices in the earlier kernels. WS-2010 might be more stable in that environment.

In a similar vein, I found far fewer problems running VMs under Vista than I did under XP. And more modern processors with the various levels of VT-x/AMD-V extensions are less prone to such mysteries than older ones.

Thanks again for all your help, I'll post again if I run into the same problem

You're welcome and good luck. (And it is always nice to get feedback:)


With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.

RIP Neil Armstrong


In reply to Re^11: PANIC: underlying join failed threded tcp server by BrowserUk
in thread PANIC: underlying join failed threded tcp server by rmahin

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others cooling their heels in the Monastery: (7)
    As of 2014-04-19 11:10 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      April first is:







      Results (480 votes), past polls