Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask
 
PerlMonks  

Re: Thread::Queue and WWW::Curl

by Corion (Pope)
on Sep 24, 2012 at 14:33 UTC ( #995399=note: print w/ replies, xml ) Need Help??


in reply to Thread::Queue and WWW::Curl

You don't show where you create instances of WWW::Curl(::Easy). I recommend that you do so separately and only within each thread, because most likely the WWW::Curl objects are backed by corresponding C objects, and those may or may not like dancing between different threads and maybe are not even threadsafe unless precautions are taken.

Personally, I would look at ditching threads and looking at the Curl-native routines in WWW::Curl::Multi. These should be able to run within a single worker thread and at least for the fetching part, you are most likely limited by network bandwidth/latency anyway.


Comment on Re: Thread::Queue and WWW::Curl

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://995399]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2015-07-06 03:56 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (70 votes), past polls