Beefy Boxes and Bandwidth Generously Provided by pair Networks
The stupid question is the question not asked
 
PerlMonks  

Re: Slow HTTP::Async responses

by bulk88 (Priest)
on Feb 15, 2013 at 00:49 UTC ( #1018820=note: print w/ replies, xml ) Need Help??


in reply to Slow HTTP::Async responses

Ive found none of the Perl HTTP modules work well in async mode. Either they aren't really synchronous (sleep timers, blocking I/O in addition to select, or things I never figured out), or they burn high amounts of CPU polling select. I use Win32::API and a small XS library to make calls to http://msdn.microsoft.com/en-us/library/windows/desktop/aa383630%28v=vs.85%29.aspx which is an internally threaded C library that uses X threads by Y select() with thread pooling design (scales to 100s of simultaneous TCPIP connections and can saturate a 100 mbps line). Look for an equivalent C library with Perl bindings on your OS. I've also used aria2c with its XML-RPC interface and reading the files off (cached in ram by OS) disk synchronously. select doesn't scale. You want to use epoll or some other next generation event system. Perl isn't very efficient on the minutia of turning packets into HTTP transactions.


Comment on Re: Slow HTTP::Async responses

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018820]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others scrutinizing the Monastery: (5)
As of 2014-07-31 22:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (254 votes), past polls