Beefy Boxes and Bandwidth Generously Provided by pair Networks Joe
Syntactic Confectionery Delight
 
PerlMonks  

Re^3: Threads and LWP::UserAgent question

by misc (Pilgrim)
on Jun 10, 2008 at 10:49 UTC ( #691201=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Threads and LWP::UserAgent question
in thread Threads and LWP::UserAgent question

Yes, there are probably too many locks.

When I wrote this module some time ago I had to struggle with deadlocks.
So I finished with locking all variables always in the same order,
also in mind it would be easier to insert new functions into the code.

When the whole thing worked, I didn't mind to clean up..

At the place you mentioned: Do you mean cond_wait would also unlock ${$self->{poolcount}} or that locking ${$self->{poolcount}} is simply unneccessary ?
I didn't trie so much to have few locks, would you say that a lock is big penalty ?

I used the hash $self everywhere because I had in mind to be able to have more than one threadpool in an script.

You are however wrong about the creating of the threads,
the module doesn't destroy any thread at all.
Instead it creates up to maxthread threads and puts the work recieved with enqueue() onto a queue.
The threads pull in T_thread() the work of the queue.

I wrote this module, because I had some problems with growing memory of long running scripts, due to creating and destroying threads.

The OP's code..
The only thing I've changed has been the small delay (select undef,undef,undef,0.2;).
Since the OP didn't say exactly what he tries to do and where all the data should go, I didn't had a closer look, instead I really did copy'n paste.

I've just been curious whether I could replace Thread::Pool with my module,
since this worked without problems I thought my implementation could be helpful.
There's also the point that the feedback, especially yours, helps me improve my code.

So, no, I don't feel you would be negative.


Comment on Re^3: Threads and LWP::UserAgent question
Re^4: Threads and LWP::UserAgent question
by gatito (Novice) on Jun 12, 2008 at 16:22 UTC
    my original code was a simple version of a spider I'm working on. It was badly written and turned out to be the wrong thing actually. So, I've been cobbling a new version together.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://691201]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (13)
As of 2014-04-16 20:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (434 votes), past polls