http://www.perlmonks.org?node_id=863785


in reply to Win32::Internet crash, XS, callbacks, perl stack & context, windows api, interpreter thread safety, 1 perl, many C threads by windows

In situations similar-but-different to this, I also have found that the best thing to use for “a callback” is a very short stub routine ... written in baggage-free “C.”   All that the callback routine does, really, is to store relevant status information and send some kind of a signal.   High-level languages such as Perl usually carry a lot of baggage around with them ... but a callback does not really need to have one.

Mutual-exclusion, of course, is quite a necessity ... but in the callback routine you use the same low-level OS primitives that you know the high-level language will also respect and use.   (You aren’t using the language’s constructs, but you make certain that the two pieces of code will interlock as they should.)

I have also learned, again quite painfully, that whenever an API provides for a “handle” (as they, of course, always do...), that handle ought to be a random number that can be safely looked-up in a hash owned by you.   In other words, it should never be “a pointer.”   Aside from the fact that we are now routinely dealing with 64-bit addresses (and 32-bit legacy integer handles), a pointer can’t be verified ... so it is a “memory stomp” waiting to happen.   Random handles, on the other hand, can be looked-up and thereby verified... and if they are no longer present in the hash structure (for whatever reason), we have “a reportable bug-condition” (probably), “but not a stomp.”

The callback logic, then, might go something like this:

  1. Grab a mutual-exclusion object (using a low-level OS call).
  2. Look up the handle that is provided.   If it cannot be found, release the mutex and exit silently.
  3. Store the information into the data-structure thus located.   This data structure should be a buffer that the high-level language side already carved out for you.   Check it for bogosity...
  4. Signal an event that the main logic might be waiting for.   Unlock the mutex and exit.

Replies are listed 'Best First'.
Re^2: Win32::Internet crash, XS, callbacks, perl stack & context, windows api, interpreter thread safety, 1 perl, many C threads by windows
by patcat88 (Deacon) on Oct 07, 2010 at 05:32 UTC
    My idea of stress testing the algorithm is removing the sleep from while not done loop in the sample script, and replacing the sleep with usleep(1);.

    I've added mutexes guarding the getting of the callback status from the global PM hash, and the calling of the Perl method that sets the status into the global PM hash from XS.

    I consider stability, FOR NOW, to be 10 runs without crashes or perl panics or strange errors on the console. 
    void WINAPI CALLBACK
PerlCallback(HINTERNET h, 
       DWORD context, 
       DWORD mystatus, 
       LPVOID mystatusinfo, 
       DWORD mystatuslength) {

    DWORD myret;
    HINTERNET myhandle;

    // Let's try with perl_call_method
    // ...to clarify:
    // a C routine 
    // called from Perl
    // callbacks this C routine 
    // that callbacks a Perl routine.
    // ;)

    // if(mystatus!=status) {
        SV **sp;
          HANDLE hMutex;
          char mutexname [sizeof("Win32InternetPerlMutex") + sizeof(DW
+ORD)*8];
          DWORD waitres;
          //char [23 + sizeof(DWORD)*8] mutexname;
        //void * myptr;
        //printf("PerlCallback: entering\n");
        printf("PerlCallback: got handle=%d context=%d mystatus=%d\n",
+h,context,mystatus);
        //printf("PerlCallback: myWin32InternetPerlInterpreterPtr=%p\n
+",myWin32InternetPerlInterpreterPtr);
        
        //comment below out to trigger original crash/bug in Win32::In
+ternet and recompile
        PERL_SET_CONTEXT(myWin32InternetPerlInterpreterPtr);
        
        //printf("PerlCallback: ThreadId=%d\n",GetCurrentThreadId());
        //printf("PerlCallback: GetLastError() =%d\n",GetLastError());
        //myptr = PERL_GET_CONTEXT;
        //printf("PerlCallback: GetLastError() =%d\n",GetLastError());
        //printf("PerlCallback: pointer from Perl_get_context() =%p\n"
+,myptr);
        //can't use dSP because we are after variable declarations
        sp = PL_stack_sp;
        ENTER;
        SAVETMPS;
        PUSHMARK(sp);
        // XPUSHs(sv_2mortal(newSVpv("Win32::Internet\0",0)));
        XPUSHs(sv_2mortal(newSViv(context)));
        XPUSHs(sv_2mortal(newSViv(mystatus)));
        switch(mystatus) {
        case INTERNET_STATUS_HANDLE_CREATED:
            myhandle=(HINTERNET) *(LPHINTERNET)mystatusinfo;
            XPUSHs(sv_2mortal(newSViv((DWORD) myhandle)));
            break;
        case INTERNET_STATUS_RESPONSE_RECEIVED:
        case INTERNET_STATUS_REQUEST_SENT:
            myret=(DWORD) *(LPDWORD)mystatusinfo;
            // printf("PerlCallback: received/sent(%d) %d bytes\n",mys
+tatus,myret);
            XPUSHs(sv_2mortal(newSViv(myret)));
            break;
        default:
            XPUSHs(sv_2mortal(newSViv(0)));
            break;
        }
        PUTBACK;
        // printf("PerlCallback: calling callback with context=%d, sta
+tus=%d\n",context,mystatus);
          sprintf(mutexname ,"Win32InternetPerlMutex%u", GetCurrentPro
+cessId());
          //printf("PerlCallback: mutexname=%s\n",mutexname);
          hMutex = OpenMutex(MUTEX_ALL_ACCESS, FALSE, mutexname);
          if (hMutex == NULL) printf("PerlCallback: OpenMutex error: %
+d\n", GetLastError() );
          else printf("PerlCallback: OpenMutex successfully opened the
+ mutex.\n");
          waitres = WaitForSingleObject(hMutex,500000);
          printf("PerlCallback: wait result =%d\n",waitres);



        perl_call_pv("Win32::Internet::callback", G_DISCARD);

          printf("PerlCallback: release mutex res=%d\n",ReleaseMutex(h
+Mutex));
          CloseHandle(hMutex);
          
        FREETMPS;
        LEAVE;

    // }
    // status=mystatus;
    // return;
}

    [download]
    was never stable until usleep(100000); 
    void WINAPI CALLBACK
PerlCallback(HINTERNET h, 
       DWORD context, 
       DWORD mystatus, 
       LPVOID mystatusinfo, 
       DWORD mystatuslength) {

    DWORD myret;
    HINTERNET myhandle;



    // Let's try with perl_call_method
    // ...to clarify:
    // a C routine 
    // called from Perl
    // callbacks this C routine 
    // that callbacks a Perl routine.
    // ;)

    // if(mystatus!=status) {
        SV **sp;
          HANDLE hMutex;
          char mutexname [sizeof("Win32InternetPerlMutex") + sizeof(DW
+ORD)*8];
          DWORD waitres;
          //char [23 + sizeof(DWORD)*8] mutexname;
        //void * myptr;
        //printf("PerlCallback: entering\n");
          sprintf(mutexname ,"Win32InternetPerlMutex%u", GetCurrentPro
+cessId());
          //printf("PerlCallback: mutexname=%s\n",mutexname);
          hMutex = OpenMutex(MUTEX_ALL_ACCESS, FALSE, mutexname);
          if (hMutex == NULL) printf("PerlCallback: OpenMutex error: %
+d\n", GetLastError() );
          else printf("PerlCallback: OpenMutex successfully opened the
+ mutex.\n");
          waitres = WaitForSingleObject(hMutex,500000);
          printf("PerlCallback: wait result =%d\n",waitres);
        printf("PerlCallback: got handle=%d context=%d mystatus=%d\n",
+h,context,mystatus);
        //printf("PerlCallback: myWin32InternetPerlInterpreterPtr=%p\n
+",myWin32InternetPerlInterpreterPtr);
        
        //comment below out to trigger original crash/bug in Win32::In
+ternet and recompile
        PERL_SET_CONTEXT(myWin32InternetPerlInterpreterPtr);
        
        //printf("PerlCallback: ThreadId=%d\n",GetCurrentThreadId());
        //printf("PerlCallback: GetLastError() =%d\n",GetLastError());
        //myptr = PERL_GET_CONTEXT;
        //printf("PerlCallback: GetLastError() =%d\n",GetLastError());
        //printf("PerlCallback: pointer from Perl_get_context() =%p\n"
+,myptr);
        //can't use dSP because we are after variable declarations
        sp = PL_stack_sp;
        ENTER;
        SAVETMPS;
        PUSHMARK(sp);
        // XPUSHs(sv_2mortal(newSVpv("Win32::Internet\0",0)));
        XPUSHs(sv_2mortal(newSViv(context)));
        XPUSHs(sv_2mortal(newSViv(mystatus)));
        switch(mystatus) {
        case INTERNET_STATUS_HANDLE_CREATED:
            myhandle=(HINTERNET) *(LPHINTERNET)mystatusinfo;
            XPUSHs(sv_2mortal(newSViv((DWORD) myhandle)));
            break;
        case INTERNET_STATUS_RESPONSE_RECEIVED:
        case INTERNET_STATUS_REQUEST_SENT:
            myret=(DWORD) *(LPDWORD)mystatusinfo;
            // printf("PerlCallback: received/sent(%d) %d bytes\n",mys
+tatus,myret);
            XPUSHs(sv_2mortal(newSViv(myret)));
            break;
        default:
            XPUSHs(sv_2mortal(newSViv(0)));
            break;
        }
        PUTBACK;
        // printf("PerlCallback: calling callback with context=%d, sta
+tus=%d\n",context,mystatus);



        perl_call_pv("Win32::Internet::callback", G_DISCARD);

        FREETMPS;
        LEAVE;
          printf("PerlCallback: release mutex res=%d\n",ReleaseMutex(h
+Mutex));
          CloseHandle(hMutex);
    // }
    // status=mystatus;
    // return;
}

    [download]
    Only 1 out of 5 runs weren't stable at usleep(10000); in the getstatus loop in the test script.

    I'm guessing that some of what BrowserUk said in Win32::Internet crash, XS, callbacks, perl stack & context, windows api, interpreter thread safety, 1 perl, many C threads by windows, in suggesting starting a new perl instance every time the C callback was fired by Windows, rather than using the main perl instance, is the "safe" way to do it.

    I'm guessing a single perl instance isn't thread safe. From my amateur looking at the XS of [threads-shared], the "shared" variables have their own perl instance that lives with them, and the "context" is switched around before using perl functions. Neither the parent/main thread or child thread perl interpreters ever get to operate on the shared variables, I think. Mutexes guard the shared interpreter from being commanded by 2 or more threads at the same time I think.

    Saving a copy of the perl instance pointer to use from another thread would only work if I can put the other, main thread to sleep, without question. I wonder if http://msdn.microsoft.com/en-us/library/ms686345%28VS.85%29.aspx would work? or is it completely unacceptable to use that in anything remotely approaching production code?
[reply]
[d/l]
[select]

      I'm looking forward to seeing sundialsvc4's detailed and insightful responses to your questions.

[reply]

In Section Seekers of Perl Wisdom