Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re: Timing web page download.

by phatWares (Initiate)
on Jul 11, 2012 at 12:07 UTC ( #981094=note: print w/ replies, xml ) Need Help??


in reply to Timing web page download.

First off, I am NO monk. But I did come into this problem a few years ago when I was building a fusker. What I ultimately did was use Parallel::Simple to branch into 4 threads with each thread using WWW:Mechanize to do the work. Since I'm on Win32 (no real pipes so to speak), I hacked up some thread control using temp files. The temp files can pass URLs to the WWW::Mechanize objects, and they can even be used to pass signals. You get into some elegant issues with concurrency and file-locks, but it CAN work.

Anyway, that was my two cents. Good luck on your project!


Comment on Re: Timing web page download.
Re^2: Timing web page download.
by Eyck (Priest) on Jul 11, 2012 at 12:14 UTC
    thanks, it looks like there is no good solutions for this.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://981094]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (8)
As of 2015-07-03 16:52 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (54 votes), past polls