comment on

Rather than creating/destroying worker threads for each file, maybe try using a queue? Each worker thread looks for items in the queue, and work on them, avoiding your sleep()s (I hate sleep in my code - it's usually a sign that I did something wrong). You can then join on all the threads at the end, which is simpler.

See the docs for Thread::Queue. You'll end up with something like this:


my $q = Thread::Queue->new();    # A new empty queue
 
# Worker threads
my @thr = map {
  threads->create(sub {
    while (my $item = $q->dequeue()) {
      # assuming undef isn't a valid value and so can be a marker.
      return unless defined $item;
      do_stuff($item, @_);
    }, $param1, $param2
   )->detach();
} 1..$thread_limit;
 
# Send work to the threads
$q->enqueue($_) for @array_of_files;

# send markers.
$q->enqueue(undef) for 1..$thread_limit;

# terminate.
$_->join() for @thr;
[download]

In this case, you can pretty much not count your main thread: after it finishes enqueueing all the files, it will only wake up once per worker thread. So, if you have a quad-core, you really can have a thread limit of 4 instead of 3. That can speed things up a bit as well. Also, you may be able to tweak it a bit - depending on how much each thread spends in I/O vs CPU, you may be able to use 5 or even more threads. Of course, you may throttle your disk at this point, so you may find your disk spinning at full tilt while your CPU usage still doesn't peg at full. At that point, yes, SSD is probably your next best bet for improving the speed.

In reply to Re: Perl Threads Boss/Worker Example by Tanktalus
in thread Perl Threads Boss/Worker Example by joemaniaci

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


There's more than one way to do things
	PerlMonks