Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Slow HTTP::Async responses

by bulk88 (Priest)
on Feb 15, 2013 at 00:49 UTC ( #1018820=note: print w/ replies, xml ) Need Help??


in reply to Slow HTTP::Async responses

Ive found none of the Perl HTTP modules work well in async mode. Either they aren't really synchronous (sleep timers, blocking I/O in addition to select, or things I never figured out), or they burn high amounts of CPU polling select. I use Win32::API and a small XS library to make calls to http://msdn.microsoft.com/en-us/library/windows/desktop/aa383630%28v=vs.85%29.aspx which is an internally threaded C library that uses X threads by Y select() with thread pooling design (scales to 100s of simultaneous TCPIP connections and can saturate a 100 mbps line). Look for an equivalent C library with Perl bindings on your OS. I've also used aria2c with its XML-RPC interface and reading the files off (cached in ram by OS) disk synchronously. select doesn't scale. You want to use epoll or some other next generation event system. Perl isn't very efficient on the minutia of turning packets into HTTP transactions.


Comment on Re: Slow HTTP::Async responses

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018820]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others drinking their drinks and smoking their pipes about the Monastery: (13)
As of 2014-12-19 08:18 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    Is guessing a good strategy for surviving in the IT business?





    Results (74 votes), past polls