srchulo has asked for the wisdom of the Perl Monks concerning the following question:
I'm using perl's threads module with a simple crawler I'm working on so I can download pages in parallel. Ocasionally, I get error messages like these:
Thread 7 terminated abnormally: read timeout at /usr/lib64/perl5/threa +ds.pm line 101. Thread 15 terminated abnormally: Can't connect to burgundywinecomp +any.com:80 (connect: timeout) at /usr/lib64/perl5/threads.pm line 101 +. Thread 19 terminated abnormally: write failed: Connection reset by + peer at /usr/lib64/perl5/threads.pm line 101.
When I run the script linearly without threads (or if I use fork() instead of threading), I do not encounter these errors. And these errors almost seem like they are from the LWP::UserAgent module that I am using, but they do not seem like they should be causing the threads to exit abnormally; they seem like errors that would be help in my HTTP::Response object. Is there some extra precaution I have to take while using perl's threads, or something I'm missing? Thanks!
UPDATEI have tracked down the source of these abnormal terminations, and it does seem to be whenever I make a request using LWP::UserAgent. If I remove the method call to download the webpage, then the errors stop.
Here is a test script that produces the error:
#!/usr/bin/perl use threads; use Thread::Queue; use LWP::UserAgent; my $THREADS=10; # Number of threads #(if you care about them) my $workq = Thread::Queue->new(); # Work to do my @stufftodo = qw(http://www.collectorsarmoury.com/ http://burgundywi +necompany.com/ http://beetreeminiatures.com/); $workq->enqueue(@stufftodo); # Queue up some work to do $workq->enqueue("EXIT") for(1..$THREADS); # And tell them when threads->create("Handle_Work") for(1..$THREADS); # Spawn our workers $_->join for threads->list; sub Handle_Work { while(my $todo=$workq->dequeue()) { last if $todo eq 'EXIT'; # All done print "$todo\n"; my $ua = LWP::UserAgent->new; my $RESP = $ua->get($todo); } threads->exit(0); }
|
---|
Replies are listed 'Best First'. | |
---|---|
Re: perl threads exiting abnormally
by zentara (Archbishop) on Nov 18, 2012 at 11:06 UTC | |
Re: perl threads exiting abnormally
by Anonymous Monk on Nov 18, 2012 at 14:40 UTC | |
by srchulo (Sexton) on Nov 18, 2012 at 23:03 UTC | |
Re: perl threads exiting abnormally
by Khen1950fx (Canon) on Nov 18, 2012 at 17:06 UTC | |
by Lotus1 (Vicar) on Nov 23, 2012 at 15:21 UTC |