Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re^4: PANIC: underlying join failed threded tcp server

by rmahin (Scribe)
on Oct 18, 2012 at 01:53 UTC ( [id://999646]=note: print w/replies, xml ) Need Help??


in reply to Re^3: PANIC: underlying join failed threded tcp server
in thread PANIC: underlying join failed threded tcp server

Alright adding that print message now.

As for adding the trace messages to threads.xs, not sure how to go about that. I searched for it in the perl installation folder for it but it was not there. Is it part of the threads.dll?

Do you think any harm would be caused from doing something like my $val = eval{$join->join()}; as just a way to prevent the server from crashing. We have multiple users issuing commands to another larger server, which randomly picks these smaller servers to issue commands against, so one of them going down is very annoying for them and requires us to constantly monitor them.

Thanks very much for your response again, you are quite the helpful monk.

Replies are listed 'Best First'.
Re^5: PANIC: underlying join failed threded tcp server
by BrowserUk (Patriarch) on Oct 18, 2012 at 02:14 UTC
    As for adding the trace messages to threads.xs, not sure how to go about that. I searched for it in the perl installation folder for it but it was not there. Is it part of the threads.dll?

    The easiest way would be to grab the latest version of threads from cpan, then build and install it manually. (Once you've checked you can still reproduce the error with the newly built version), you can then modify threads.xs, re-build and install.

    Do you think any harm would be caused from doing something like my $val = eval{$join->join()}; as just a way to prevent the server from crashing.

    If you are not doing anything with the return value from the join, it would probably be an okay temporary workaround. If you're only joining teh thread to make it go away, and it is going away of its own accord, in one sense, job done.

    But in the long term, whatever the cause might just be a symptom of a deeper issue in your code; or in the threads module; or perl itself. As you have a recreatable scenario, it would be silly not to use it to help track down the underlying issue.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    RIP Neil Armstrong

    p
      Hey thanks for explanation of how to do that. I got what you asked for, and here is the link to a text file of the output from the start to finish https://dl.dropbox.com/u/19686501/serverOutput.txt None of that info is particularly helpful to me at least but if you see something I don't, I'm all ears. Thanks for your patience in me getting back to you. Took a while to recreate this time.
        None of that info is particularly helpful to me at least but if you see something I don't, I'm all ears.

        What I can see from that info is:

        1. At line 3426:
          RXD+ (982) > Command completed with a return code of 0 RXD+ (0) > _handle() output: 51686792 thread handle:2d00 thread-id: 4240x

          A thread, with a perl thread Id (tid) of 982 completes and Windows thread handle:2d00 has (just prior to joining) a OS thread ID of 4240, and the join completes without error.

        2. Later, at line 3775:
          thread handle:2d00 thread-id: 0x GetLastError() output: '6' Join failed with 'Bad file descriptor' : 'The handle is invalid' at rxd.pl line 1128.

          Just prior to a join attempt, 'another thread' with the same OS thread handle 2d00, this time does not have an OS thread id, which indicates that the thread handle:2d00 is indeed an invalid handle as the system reports.

        What that indicates is that either:

        • The OS is reusing the same OS thread handle -- which whilst possible seems unlikely.
        • Or this; threads->list(threads::joinable) is returning the handle of an already joined thread. Which also seems unlikely, but could happen if the (Perl) internal linked list got corrupted some how.

        The next thing I would try is adding a similar trace line at the end of S_ithread_create(), something like:

        S_ithread_create( ... printf( "ITCREATE: thread handle:%x thread-id: %dx\n", thread->han +dle, GetThreadId( thread->handle ) ); MY_POOL.running_threads++; return (thread); }

        And also in

        STATIC void S_ithread_free(pTHX_ ithread *thread) { ... #ifdef WIN32 printf( "ITFREE: thread handle:%x thread-id: %dx\n", thread->handl +e, GetThreadId( thread->handle ) ); if (handle) { CloseHandle(handle); } #endif ... }

        The idea is to isolate whether -- when the error occurs -- the invalid handle is to a thread that has already been freed -- in which case the bug is in threads::list() -- or to a thread that has not yet been freed -- in which case it would mean an OS error of some kind; perhaps resource constraint;

        I breifly looked at trying to run your server here and trying to re-create the failure. Whilst the server runs and accepts connections from a telnet seesion, it won't accept input from it because (my) telnet sends character by character and it is expecting entire commands wrapped in your (incredibly complicated) comms protcol.

        There is no way I am going to be able to reverse engineer a client that can talk that protocol.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

        Sight of the changes made to the OP code and threads.xs would make interpreting that output a possibility :)

        Also, what OS/version; Perl/version; threads/version?


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        RIP Neil Armstrong

Re^5: PANIC: underlying join failed threded tcp server
by Anonymous Monk on Oct 18, 2012 at 02:02 UTC

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://999646]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others meditating upon the Monastery: (4)
As of 2024-04-19 21:39 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found