Beefy Boxes and Bandwidth Generously Provided by pair Networks
P is for Practical
 
PerlMonks  

Threads and HTTPS Segfault

by onelesd (Pilgrim)
on Aug 21, 2011 at 06:25 UTC ( #921497=perlquestion: print w/ replies, xml ) Need Help??
onelesd has asked for the wisdom of the Perl Monks concerning the following question:

I have a script in which I use threads; use LWP; to make many simultaneous HTTPS connections using LWP::UserAgent. If there is more than one thread making a request at the same time, threads will crash and/or perl will segfault. Most of the time in each thread is spent making the HTTPS request, so locking while the request is made defeats my purpose for using threads and is not an option.

I traced down the problem to Crypt::SSLeay not being thread-safe, which LWP uses to make SSL requests. Here's the bug report. There is also a very old thread here at perlmonks without a solution.

How can I work around this?

$ perl -v This is perl 5, version 12, subversion 3 (v5.12.3) built for x86_64-li +nux-gnu-thread-multi

Comment on Threads and HTTPS Segfault
Select or Download Code
Re: Threads and HTTPS Segfault
by Anonymous Monk on Aug 21, 2011 at 06:33 UTC

    How can I work around this?

    By using require instead of use, to only load LWP (and any other non-thread safe modules) in child threads only

      I seriously doubt that will help in this case. The non-theadsafe part is most likely to be in one of the XS components or their underlying C libraries and will likely manifest themselves regardless of whether the Perlish code is used then cloned into the threads or required into each thread separately.

      Working around module non-thread-safety by requireing only really works if you then only make use of the module from a single thread, which wouldn't achieve the OPs goal.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
      Thanks for this suggestion, but it didn't work. I used Thread::Use inside the threads to only load LWP there and still encountered the bug.
Re: Threads and HTTPS Segfault
by eyepopslikeamosquito (Canon) on Aug 21, 2011 at 07:45 UTC

    How can I work around this?
    Is switching from threads to processes an option? Have you considered POE or AnyEvent?

      I use Thread::Queue; to async {$Qin->enqueue($_) while (<>);}; and the child threads loop and block while (my $line = $Qin->dequeue) {...}. I am not familiar with POE or AnyEvent - is there a similar pattern using those modules?
Re: Threads and HTTPS Segfault
by moritz (Cardinal) on Aug 21, 2011 at 08:04 UTC
Re: Threads and HTTPS Segfault
by Anonymous Monk on Aug 21, 2011 at 08:06 UTC

    I traced down the problem to Crypt::SSLeay not being thread-safe, which LWP uses to make SSL requests. Here's the bug report. There is also a very old thread here at perlmonks without a solution.

    The bug report contains a patch, try applying it :)

      I am not familiar with C, and the patch is for an old version. Here is the patch, if anyone in the know cares to comment. There are a couple of notes about the patch in the bug report which I don't quite understand: working only with pthreads and a possible problem with one part of the patch ("casting a 'pthread_t' to an 'unsigned long'".
      diff -r -c Crypt-SSLeay-0.57_01/META.yml Crypt-SSLeay-0.57_01+ithreads +/META.yml *** Crypt-SSLeay-0.57_01/META.yml 2008-02-18 23:43:14.000000000 +09 +00 --- Crypt-SSLeay-0.57_01+ithreads/META.yml 2008-11-13 09:53:09.0000 +00000 +0900 *************** *** 1,6 **** --- #YAML:1.0 name: Crypt-SSLeay ! version: 0.57_01 abstract: OpenSSL support for LWP license: perl author: --- 1,6 ---- --- #YAML:1.0 name: Crypt-SSLeay ! version: 0.57_01+ithreads abstract: OpenSSL support for LWP license: perl author: diff -r -c Crypt-SSLeay-0.57_01/SSLeay.pm Crypt-SSLeay-0.57_01+ithread +s/SSLeay.pm *** Crypt-SSLeay-0.57_01/SSLeay.pm 2008-02-18 23:33:14.000000000 +0 +900 --- Crypt-SSLeay-0.57_01+ithreads/SSLeay.pm 2008-11-13 09:51:28.000 +000000 +0900 *************** *** 2,8 **** use strict; use vars '$VERSION'; ! $VERSION = '0.57_01'; eval { require XSLoader; --- 2,8 ---- use strict; use vars '$VERSION'; ! $VERSION = '0.57_01+ithreads'; eval { require XSLoader; diff -r -c Crypt-SSLeay-0.57_01/SSLeay.xs Crypt-SSLeay-0.57_01+ithread +s/SSLeay.xs *** Crypt-SSLeay-0.57_01/SSLeay.xs 2008-02-18 23:11:15.000000000 +0 +900 --- Crypt-SSLeay-0.57_01+ithreads/SSLeay.xs 2008-11-13 11:37:55.000 +000000 +0900 *************** *** 18,23 **** --- 18,24 ---- #define PERL5 1 #endif + #define OPENSSL_THREAD_DEFINES /* ssl.h or openssl/ssl.h is included from the crypt_ssleay_version * file which is written when building with perl Makefile.PL * #include "ssl.h" *************** *** 25,34 **** --- 26,73 ---- #include "crypt_ssleay_version.h" #undef Free /* undo namespace pollution from crypto.h */ + + #if defined(OPENSSL_THREADS) + #include <pthread.h> + #endif #ifdef __cplusplus } #endif + #if defined(OPENSSL_THREADS) + static pthread_mutex_t *mutex_buf = NULL; + static int mutex_use = 0; + static int mutex_free = 0; + static pthread_mutex_t mutex_lib; + + /** + * OpenSSL locking function. + * + * @param mode lock mode + * @param n lock number + * @param file source file name + * @param line source file line number + * @return none + */ + static void locking_function(int mode, int n, const char *file, int +line) + { + if (mode & CRYPTO_LOCK) { + pthread_mutex_lock(&mutex_buf[n]); + } else { + pthread_mutex_unlock(&mutex_buf[n]); + } + } + + /** + * OpenSSL uniq id function. + * + * @return thread id + */ + static unsigned long id_function(void) + { + return ((unsigned long) pthread_self()); + } + #endif // OPENSSL_THREADS /* moved this out to Makefile.PL so user can * see value being used printed during build *************** *** 103,109 **** --- 142,171 ---- static int bNotFirstTime; char buf[1024]; int rand_bytes_read; + int i; + #if defined(OPENSSL_THREADS) + if( mutex_use == 0 ) + { + mutex_use++; + pthread_mutex_init( &mutex_lib, NULL ); + pthread_mutex_lock( &mutex_lib ); + /* static locks area */ + mutex_buf = malloc(CRYPTO_num_locks() * sizeof(pthread_m +utex_t)); + //if (mutex_buf == NULL) { + // return (-1); + //} + for (i = 0; i < CRYPTO_num_locks(); i++) { + pthread_mutex_init(&mutex_buf[i], NULL); + } + /* static locks callbacks */ + CRYPTO_set_locking_callback(locking_function); + CRYPTO_set_id_callback(id_function); + } else { + pthread_mutex_lock(&mutex_lib); + mutex_use++; + } + #endif // OPENSSL_THREADS if(!bNotFirstTime) { SSLeay_add_all_algorithms(); SSL_load_error_strings(); *************** *** 136,147 **** --- 198,214 ---- SSL_CTX_set_default_verify_paths(ctx); SSL_CTX_set_verify(ctx, SSL_VERIFY_NONE, NULL); RETVAL = ctx; + #if defined(OPENSSL_THREADS) + pthread_mutex_unlock(&mutex_lib); + #endif OUTPUT: RETVAL void SSL_CTX_free(ctx) SSL_CTX* ctx + CODE: + mutex_free++; int SSL_CTX_set_cipher_list(ctx, ciphers)
      I applied the patch, the build and test went fine, but using the module causes a crash. It's beyond me.
      *** glibc detected *** /perlbrew/perls/perl-5.12.3/bin/perl: free(): i +nvalid next size (fast): 0x00007f997c0a5ad0 *** ======= Backtrace: ========= /lib/libc.so.6(+0x775b6)[0x7f998f8ea5b6] /lib/libc.so.6(+0x7dbfb)[0x7f998f8f0bfb] /lib/libc.so.6(realloc+0xf0)[0x7f998f8f10b0] /usr/lib/libcrypto.so.0.9.8(CRYPTO_realloc+0x5f)[0x7f99855fafdf] /usr/lib/libcrypto.so.0.9.8(lh_insert+0x177)[0x7f998565acd7] /usr/lib/libcrypto.so.0.9.8(+0xcf259)[0x7f998565f259] /usr/lib/libcrypto.so.0.9.8(ERR_load_strings+0x41)[0x7f998565ea11] /usr/lib/libcrypto.so.0.9.8(ERR_load_ERR_strings+0x7c)[0x7f998565fabc] /usr/lib/libcrypto.so.0.9.8(ERR_load_crypto_strings+0x9)[0x7f998565f57 +9] /usr/lib/libssl.so.0.9.8(SSL_load_error_strings+0x9)[0x7f9985956729] /perlbrew/perls/perl-5.12.3/lib/site_perl/5.12.3/x86_64-linux-gnu-thre +ad-multi/auto/Net/SSLeay/SSLeay.so(XS_Net__SSLeay_load_error_strings+ +0x73)[0x7f9985baa323]/perlbrew/perls/perl-5.12.3/lib/5.12.3/x86_64-li +nux-gnu-thread-multi/CORE/libperl.so(Perl_pp_entersub+0x530)[0x7f9990 +99f460]
Re: Threads and HTTPS Segfault
by Khen1950fx (Canon) on Aug 21, 2011 at 08:09 UTC
    Instead of using threads, use coros. LWP::Protocol::AnyEvent::http makes it easy. Just start off like:
    #!/usr/bin/perl use strict; use warnings; use LWP::Protocol::AnyEvent::http; use Coro qw( async ); my $ua = LWP::UserAgent->new(); $ua->protocols_allowed([qw( http https )]);
      Coro doesn't scale across multiple processors and that would hurt performance, but I suppose I could move the HTTPS request from the child threads into a Coro thread in the parent, but I'd need some way to send data from the children to the Coro thread.

        If you cannot fix the problem and need bi-di communications with the kids, then you could do worse than to use a queue talking to threads that run lwp-requests via piped-opens. The https sessions run in separate processes, but you fetch the data back into the parent process where you can coordinate between them.


        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        You say in your original post, that...

        Most of the time in each thread is spent making the HTTPS request ...

        If this is true you shouldn't lose much performance by using an event framework and/or Coro... depending on what you mean by "most" I guess...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://921497]
Approved by GrandFather
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (5)
As of 2014-09-17 21:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (99 votes), past polls