Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re^4: threads: work crew memory leak

by rakzer (Novice)
on Oct 17, 2010 at 09:59 UTC ( #865789=note: print w/ replies, xml ) Need Help??


in reply to Re^3: threads: work crew memory leak
in thread threads: work crew memory leak

FWIW, the reason why I originally chose to create that many threads is that threads could (and quite frequently will) time out due to network related issues. Creating a timer thread to kill timed out threads seemed very easy and handy. Creating a fork within a thread with a signal handler killing the fork on the other hand would be a lot more expensive.


Comment on Re^4: threads: work crew memory leak
Re^5: threads: work crew memory leak
by BrowserUk (Pope) on Oct 17, 2010 at 17:13 UTC
    Creating a timer thread to kill timed out threads seemed very easy and handy.

    That sounds simple and convenient, but threads have state. And if you go around just killing them, you don't give them the opportunity to clean up after themselves and that's where 'mysterious' memory leaks arise.

    Creating a fork within a thread with a signal handler killing the fork on the other hand would be a lot more expensive.

    I certainly agree with you that that is not a viable option.

    But there are better alternatives to both those approaches. The details depend upon the type of "network related issues" you are concerned about?

    In general, using cancellable asynchronous IO or non-blocking IO, within the thread, to allow you to abandon slow or broken connections, is a viable alternative. It allows you to retain the benefits of threading: simplicity of 'serial flow' architecture; scalability across cores; whilst avoiding the traps of 'throwing threads at the problem' & the difficulties of clean-up associated with abandoning threads.

    You gain the benefits of a non-blocking IO, without the need to invert the flow of control, or have the need for abandonable IO to dictate the architecture of your entire application.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      That sounds simple and convenient, but threads have state. And if you go around just killing them, you don't give them the opportunity to clean up after themselves and that's where 'mysterious' memory leaks arise.
      What would be the right way(tm) to kill a hanging thread as far as proper garbage collection goes?
      But there are better alternatives to both those approaches. The details depend upon the type of "network related issues" you are concerned about?
      Basically all the program does is retrieving the RFC 2616 status code of a large number (+500k) of distfiles from various different http/ftp servers using HTTP::Request and LWP::UserAgent.

      Setting $ua->timeout(10); deosn't seem to be working at all. I'm now doing:

      my $code; eval { local $SIG{ALRM} = sub {die "alarm\n"}; alarm $defcfg->{'link_check_timeout'}; # LWP stuff $code= ...; alarm 0; }
      Not sure if that's the best thing but it seems to work better than setting an LWP timeout.
        What would be the right way(tm) to kill a hanging thread as far as proper garbage collection goes?

        There is no right way to kill a thread.

        Even at the OS level, the TerminateThread() api--if your OS has one--is designated an action of extreme last resort:

        Remarks

        TerminateThread is used to cause a thread to exit. When this occurs, the target thread has no chance to execute any user-mode code. DLLs attached to the thread are not notified that the thread is terminating. The system frees the thread's initial stack.

        Windows Server 2003 and Windows XP/2000: The target thread's initial stack is not freed, causing a resource leak.

        TerminateThread is a dangerous function that should only be used in the most extreme cases. You should call TerminateThread only if you know exactly what the target thread is doing, and you control all of the code that the target thread could possibly be running at the time of the termination.

        And that's from C. From Perl, you simply have no way of meeting those requirements.

        Not sure if that's the best thing but it seems to work better than setting an LWP timeout.

        The LWP timeout is a generic setting that it tries to apply across a wide range of protocols and phases of the communication process. But during socket operations there are many phases. The initial connection; the ack/nak responses; the transmission of blocks of content and the associated handshaking; the shutdown etc.

        If, for example, you are talking to a slow or busy server, it is quite possible for each phase to take just less than the timeout value you set. With the result that the overall time for the communication can be several time longer than the timeout value. Hence, you are left wondering why did it continue for 30 or 40 seconds when I set a 10 seconds timeout? And the answer is because no individual stage of the communication broached the 10 second limit.

        If your use of alarm is meeting your requirements, then it is as good as any other way. I've used it myself successfully, but I have found that there appear to be some points in the process of communicating with a server that it won't successfully interrupt due to "safe signals".

        If you find that you sometimes get servers that cause your threads to hang around blocked for longer than is desirable, despite your use of alarm, then you might want to try adding Perl::Unsafe::Signals to your eval block. See the module's synopsis for how.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://865789]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (6)
As of 2014-09-15 10:14 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (146 votes), past polls