Beefy Boxes and Bandwidth Generously Provided by pair Networks
good chemistry is complicated,
and a little bit messy -LW
 
PerlMonks  

Slow WWW::Mechanize->get

by AnK_ (Initiate)
on Jan 25, 2013 at 22:02 UTC ( #1015419=perlquestion: print w/ replies, xml ) Need Help??
AnK_ has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks, I spent considerable time trying to figure out why the following code works fast on some sites and very slow on others. All sites are equally fast in Firefox.

my $start_page = 'https://www.yahoo.com';#fast #$start_page = 'https://github.com'; #fast #$start_page = 'https://metacpan.org'; #fast my $mech = WWW::Mechanize->new()->get( $start_page );

Replacing WWW::Mechanize with WWW::Mechanize::Firefox corrects the problem. However requires Firefox I am trying to avoid. I attempted to analyze 'get' request in Firefox using Tamper Data but could not figure out what is the difference. At this point I kindly request your wisdom. Please be specific as much as posiible - I am not an expert in programming in general and in perl specifically. Thank you in advance.

Comment on Slow WWW::Mechanize->get
Download Code
Re: Slow WWW::Mechanize->get
by roboticus (Canon) on Jan 26, 2013 at 00:54 UTC

    Ank_:

    The only theory I have is that the site is looking at the user agent string to do some things, and when it doesn't contain a 'mainstream' agent string, their site may be doing something to make it slower. To test the theory, try setting up a different agent string and see if it replies faster.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      Thanks, roboticus, will do ...
      Dear roboticus, I just recognized the right question to ask: How can I print/see/dump/compare WWW::Mechanize's and Firefox's user agent strings set to the site.
        read the docs :) ->res->headers->as_string

        From the docs:

        $mech->agent_alias('Windows Mozilla') is probably sufficient. You can use $mech->known_agent_aliases() to get a list of the agent aliases built into WWW::Mechanize.

        ...roboticus

        When your only tool is a hammer, all problems look like your thumb.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1015419]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others perusing the Monastery: (9)
As of 2014-08-23 18:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (178 votes), past polls