Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Slow WWW::Mechanize->get

by AnK_ (Initiate)
on Jan 25, 2013 at 22:02 UTC ( #1015419=perlquestion: print w/ replies, xml ) Need Help??
AnK_ has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks, I spent considerable time trying to figure out why the following code works fast on some sites and very slow on others. All sites are equally fast in Firefox.

my $start_page = 'https://www.yahoo.com';#fast #$start_page = 'https://github.com'; #fast #$start_page = 'https://metacpan.org'; #fast my $mech = WWW::Mechanize->new()->get( $start_page );

Replacing WWW::Mechanize with WWW::Mechanize::Firefox corrects the problem. However requires Firefox I am trying to avoid. I attempted to analyze 'get' request in Firefox using Tamper Data but could not figure out what is the difference. At this point I kindly request your wisdom. Please be specific as much as posiible - I am not an expert in programming in general and in perl specifically. Thank you in advance.

Comment on Slow WWW::Mechanize->get
Download Code
Replies are listed 'Best First'.
Re: Slow WWW::Mechanize->get
by roboticus (Chancellor) on Jan 26, 2013 at 00:54 UTC

    Ank_:

    The only theory I have is that the site is looking at the user agent string to do some things, and when it doesn't contain a 'mainstream' agent string, their site may be doing something to make it slower. To test the theory, try setting up a different agent string and see if it replies faster.

    ...roboticus

    When your only tool is a hammer, all problems look like your thumb.

      Dear roboticus, I just recognized the right question to ask: How can I print/see/dump/compare WWW::Mechanize's and Firefox's user agent strings set to the site.

        From the docs:

        $mech->agent_alias('Windows Mozilla') is probably sufficient. You can use $mech->known_agent_aliases() to get a list of the agent aliases built into WWW::Mechanize.

        ...roboticus

        When your only tool is a hammer, all problems look like your thumb.

        read the docs :) ->res->headers->as_string
      Thanks, roboticus, will do ...

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1015419]
Approved by ww
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (6)
As of 2015-07-31 02:51 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The top three priorities of my open tasks are (in descending order of likelihood to be worked on) ...









    Results (274 votes), past polls