Re: AnyEvent Parallel HTTP

[Yary]

AnyEvent is awesome, but it doesn't fork or thread. It's for co-operative event handling within a single process. You should look at MCE, Parallel::Async, IPC::Fork::Simple, Parallel::ForkManager, Thread::Pool::Simple- something like that, which actually opens a new process or thread to do work in parallel (perhaps on separate CPUs, if you have them)

EDIT Or look more at AnyEvent's docs to see how it can be made to fork or thread using any of these: AnyEvent::Fork, AnyEvent::Fork::RPC, AnyEvent::Fork::Pool, AnyEvent::Fork::Remote, Coro::AnyEvent

[Corion]

Uh - AnyEvent is a framework to do tasks in parallel that are not CPU bound. And usually, web scraping is not CPU bound. Distributing the load across more than one CPU makes sense if your load is CPU bound, but not necessarily when scraping a website over the network.

There is no need to involve threads or forking when using AnyEvent.