Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Re: Slow HTTP::Async responses

by bulk88 (Priest)
on Feb 15, 2013 at 00:49 UTC ( #1018820=note: print w/replies, xml ) Need Help??

in reply to Slow HTTP::Async responses

Ive found none of the Perl HTTP modules work well in async mode. Either they aren't really synchronous (sleep timers, blocking I/O in addition to select, or things I never figured out), or they burn high amounts of CPU polling select. I use Win32::API and a small XS library to make calls to which is an internally threaded C library that uses X threads by Y select() with thread pooling design (scales to 100s of simultaneous TCPIP connections and can saturate a 100 mbps line). Look for an equivalent C library with Perl bindings on your OS. I've also used aria2c with its XML-RPC interface and reading the files off (cached in ram by OS) disk synchronously. select doesn't scale. You want to use epoll or some other next generation event system. Perl isn't very efficient on the minutia of turning packets into HTTP transactions.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1018820]
[davido]: jdporter : Is there a specific issue, or are we all just getting up there?

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (9)
As of 2018-06-21 16:40 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (118 votes). Check out past polls.