httptech has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to understand the Thread module better. I'm working on a test script that uses 5 worker threads at a time to download web content from 10 URLs. The first 5 threads start fine, but then after the first URL is retrieved the program hangs.

I've tried setting it up several different ways, but I'm just not getting anything that works yet.

I have been able to successfully write a script that just starts one thread per URL that gets all the sites content in parallel, but I want to be able to limit the number of threads running at any given time in order to keep the script from sucking up all the bandwidth and CPU time as the list of URLs grows.

Here is my non-working example:

use Thread; use Thread::Queue; use LWP::Simple; use strict; my $q = new Thread::Queue; $q->enqueue qw( ); while ($q->pending) { my $kid; my @threads = Thread->list; my $current = scalar(@threads); if ($current < 6) { my $url = $q->dequeue; print "Retrieving $url\n"; $kid = new Thread(\&get_url, $url); } } $q->enqueue(undef); # Don't know if I even need this part; it has no effect at this point for (Thread->list) { if ($_->tid && !Thread::equal($_, Thread->self)) { $_->join; } } sub get_url { my $url = shift; get($url); print "Retrieved $url\n"; }

Replies are listed 'Best First'.
RE: Threading
by eduardo (Curate) on May 19, 2000 at 22:27 UTC
    hi! this isn't really the answer you are looking for, however... PLEASE UNDER NO CIRCUMSTANCES USE THREADS UNDER PERL! :) hi! sorry about that... perl threads are not stable under any circumstances (actually, even the code in perlthrtut breaks!) so, use fork, please!
      They are of questionable stability.

      Our company has multi-threaded Perl code as a midleware server (Java/Perl/SQL) used in a production environment, no less. As long as we do not run the server SMP (it really hurts to have two CPUs but only use one...), we have no problems.

      Yes, we are considerably daft to have done such a thing (over a year ago, now), but it works well, in the right conditions.


      What can I say, I like to live on the bleeding edge sometimes. I realize the threading code is still experimental even in Perl 5.6.0, and that's exactly what I'm doing with it, experimenting. :)

      Unfortunately it doesn't look like anyone else has experimented with it enough to answer my question. And I thought it would probably be a simple answer. :(

Re: Threading
by linux454 (Pilgrim) on Apr 21, 2001 at 19:47 UTC
    What I would do is put the $q->dequeue; down in the get_url subroutine;
    such that:
    sub get_url { my $url = $q->dequeue); get($url); print "Retrieved $url\n"; }
    I'm using threads to act similar to a select or fileevent
    in Perl/Tk to monitor log files so I'm in the same
    boat you are in trying to learn some more about Perl Threads
    But I would look into your join code and see what you can see.
    Is the script holding up at Thread creation or joining?