Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

Re: Timing web page download.

by phatWares (Initiate)
on Jul 11, 2012 at 12:07 UTC ( #981094=note: print w/ replies, xml ) Need Help??


in reply to Timing web page download.

First off, I am NO monk. But I did come into this problem a few years ago when I was building a fusker. What I ultimately did was use Parallel::Simple to branch into 4 threads with each thread using WWW:Mechanize to do the work. Since I'm on Win32 (no real pipes so to speak), I hacked up some thread control using temp files. The temp files can pass URLs to the WWW::Mechanize objects, and they can even be used to pass signals. You get into some elegant issues with concurrency and file-locks, but it CAN work.

Anyway, that was my two cents. Good luck on your project!

Replies are listed 'Best First'.
Re^2: Timing web page download.
by Eyck (Priest) on Jul 11, 2012 at 12:14 UTC
    thanks, it looks like there is no good solutions for this.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://981094]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others lurking in the Monastery: (4)
As of 2016-06-27 04:09 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    My preferred method of making French fries (chips) is in a ...











    Results (335 votes). Check out past polls.