Beefy Boxes and Bandwidth Generously Provided by pair Networks
"be consistent"

Slow WWW::Mechanize->get

by AnK_ (Initiate)
on Jan 25, 2013 at 22:02 UTC ( #1015419=perlquestion: print w/replies, xml ) Need Help??
AnK_ has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks, I spent considerable time trying to figure out why the following code works fast on some sites and very slow on others. All sites are equally fast in Firefox.

my $start_page = '';#fast #$start_page = ''; #fast #$start_page = ''; #fast my $mech = WWW::Mechanize->new()->get( $start_page );

Replacing WWW::Mechanize with WWW::Mechanize::Firefox corrects the problem. However requires Firefox I am trying to avoid. I attempted to analyze 'get' request in Firefox using Tamper Data but could not figure out what is the difference. At this point I kindly request your wisdom. Please be specific as much as posiible - I am not an expert in programming in general and in perl specifically. Thank you in advance.

Replies are listed 'Best First'.
Re: Slow WWW::Mechanize->get
by roboticus (Chancellor) on Jan 26, 2013 at 00:54 UTC


    The only theory I have is that the site is looking at the user agent string to do some things, and when it doesn't contain a 'mainstream' agent string, their site may be doing something to make it slower. To test the theory, try setting up a different agent string and see if it replies faster.


    When your only tool is a hammer, all problems look like your thumb.

      Dear roboticus, I just recognized the right question to ask: How can I print/see/dump/compare WWW::Mechanize's and Firefox's user agent strings set to the site.

        From the docs:

        $mech->agent_alias('Windows Mozilla') is probably sufficient. You can use $mech->known_agent_aliases() to get a list of the agent aliases built into WWW::Mechanize.


        When your only tool is a hammer, all problems look like your thumb.

        read the docs :) ->res->headers->as_string
      Thanks, roboticus, will do ...

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1015419]
Approved by ww
[erix]: Nancy-Bari: 1320 km ... He'll need some time :)
[LanX]: he was "meating" Italian monks oO?
[erix]: then again, there are people who do 10 full triathlons on 10 consecutive days, so it must be doable
choroba's friend has done the Pacific Crest Trail, it's 4200 km
LanX Ladies and gentlemen: due to shortages of "Body of Christ" we have to resort to Body of Monks ...
LanX ... buon appetito!

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (6)
As of 2017-12-13 15:10 GMT
Find Nodes?
    Voting Booth?
    What programming language do you hate the most?

    Results (369 votes). Check out past polls.