Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"
 
PerlMonks  

Re^2: Multithreaded Server Crashes under heavy load

by rmahin (Beadle)
on Aug 29, 2012 at 17:29 UTC ( #990514=note: print w/ replies, xml ) Need Help??


in reply to Re: Multithreaded Server Crashes under heavy load
in thread Multithreaded Server Crashes under heavy load

Made the change to use threads but it produced the same result. Looking for a tool to use to capture the information requested now. Is there a tool that you could recommend that can capture all of that information on a Windows environment? Thanks.


Comment on Re^2: Multithreaded Server Crashes under heavy load
Re^3: Multithreaded Server Crashes under heavy load
by BrowserUk (Pope) on Aug 29, 2012 at 17:40 UTC
    Is there a tool that you could recommend that can capture all of that information on a Windows environment?

    Yes. Type perfmon.exe into a command line.

    There is a bit of a learning curve involved in using it, but it is gui with pretty extensive help built in.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

      Ok, so heres the link to the comma delimited CSV file perfmon generated. Did not see how to get it to generate socket information for a specific process though. Also, the thing doesn't show that perl ever reached 100% cpu usage, but using the task manager shows that it spawns multiple perl processes, and that cpu usage is maxed for a while, eventually all but one perl process disappears from the task manager, cpu usage drops back down, and then the program is unresponsive.

      https://dl.dropbox.com/u/19686501/RXD_000006.csv

      Made this log by adding a new counter log, and then adding various process counters. If this was incorrect let me know.

        Posted anonymously because my account is being f**** with (permission denied) -- BrowserUk

        Did not see how to get it to generate socket information for a specific process though.

        If you go into the Performance monitor view, click the green + icon, scroll the top-left list and expand the TCPv4 topic, you can select various counters relating to TCP. The most interesting counters are "Connections Active/passive/established/reset". The trace is most useful if there is not to much other TCP traffic on the box as it isn't process specific.

        Alternatively, you could grab TCPView which is a little more intuitive for viewing the instantaneous state, but not so good for tracing over an extended period.

        For an instantaneous snapshot in a textual form, try netstat -a -b and redirect to a file.

        The log you posted only covers a 3 minute period, and there are already 301 threads running when it starts. None of them apparently finish, so the trace essentially shows a static state. No so useful. I can only assume that by the time that trace is started, everything is hung.

        Without seeing threads being started and ending it is pretty useless. You need to establish whether threads are ever ending? (Perhaps your debug log shows that?) And whether once they end, whether they are actually going away.

        I suspect that either your connections are not being closed properly and are taking a long time to time out. And/or your threads ending, but not going away. I cannot (yet) see any reason from the source code however, and the log doesn't really tell me anything.

        The elapsed time suggests that this snapshot was taken 38 hours into a run? You really need to start PerfMon immediately the server is started and monitor its cpu/threads/memory/sockets at (say) 1 minute intervals until it appears to hang. Running netstat to a file with a similar interval over the same run would give the best information.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://990514]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others cooling their heels in the Monastery: (12)
As of 2014-07-24 13:05 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (160 votes), past polls