Beefy Boxes and Bandwidth Generously Provided by pair Networks
Welcome to the Monastery
 
PerlMonks  

Re^2: Scraping using WWW::Mechanize::Firefox

by tobyink (Abbot)
on Feb 06, 2013 at 23:56 UTC ( #1017529=note: print w/ replies, xml ) Need Help??


in reply to Re: Scraping using WWW::Mechanize::Firefox
in thread Scraping using WWW::Mechanize::Firefox

If you visit the link in your browser you'll see the page contains:

1-20 of 1,975 results

I assume that this is the source of the "1975" which the OP wants to extract.

package Cow { use Moo; has name => (is => 'lazy', default => sub { 'Mooington' }) } say Cow->new->name


Comment on Re^2: Scraping using WWW::Mechanize::Firefox
Re^3: Scraping using WWW::Mechanize::Firefox
by census (Initiate) on Feb 07, 2013 at 00:57 UTC
    yes, that is clearly the 1975 I am looking for. And by does not work, I am not getting 1975. The code would work if it gave me 1975 in legit way for that test case.
Re^3: Scraping using WWW::Mechanize::Firefox
by ww (Bishop) on Feb 07, 2013 at 01:11 UTC
    Now that tobyink has -- in effect, :-) -- vouched for that site and specific address, I've been trying to get there too.

    That raises a new issue: when I connected (and it's an LDS geneaology site, for all concerned) the query ran a very long time -- more than 3 minutes of rotating arrow without any timeout or other informational message -- each of three times I tried to follow it.

    I don't have superfast DSL, but it's not that slow...and so I wonder if the problem may be in the length of the (multiply-compounded) query or in the OP's connection.

      I've updated the code. Now I'm stuck in a different loop as I noted. On my connection and computer, it takes less then 10 seconds to get the site loaded. It was probably less than 2.5 secs. However, at the moment, perl is going through that infinite loop of printing sleep for way more than at least 1 minute.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1017529]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (10)
As of 2014-09-02 08:20 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (20 votes), past polls