Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Re: suck on www::mechanize question

by smackdab (Pilgrim)
on Mar 01, 2004 at 07:10 UTC ( #332809=note: print w/ replies, xml ) Need Help??


in reply to Re: suck on www::mechanize question
in thread can't get www::mechanize to work on a web site

Thanks for trying it!
The connection refused error might be the problem, but I can get it to work in the browser, not www::mechanize. I'll look into HTTP::Recorder and see what it does...

Did you get it to work??????

I tried switching the artist/song to others w/o change. I also tried the ->follow_link having the artist and song as part of the search string, but no difference.

The original $mech->get() has artist and song, if you know of any artist/song that works, maybe that would give me a hint...(or maybe not...)


Comment on Re: Re: suck on www::mechanize question
Re: Re: Re: suck on www::mechanize question
by leira (Monk) on Mar 01, 2004 at 07:33 UTC
    OK, further investigation suggests that it's not one bad link, but that the server just sometimes returns a 500 error. Other times it succeeds. If I run my script several times, it will sometimes succeed and sometimes fail -- and I get the same results if I try to follow the link several times in my browser.

    I was able to get around it like this:

    my $maxtries = 10; my $i = 1; while ($i <= $maxtries) { $mech->follow_link(text => "Hey Jude (lennon/mccartney)", n => "1" +); last if $mech->success; $mech->back(); $i++; } $mech->success() or die "Can't find song page\n";

    Of course, you can set $maxtries to whatever you think is prudent, and you can put in a sleep() in the loop if you think that might help.

    Linda

      Thanks again for the help

      I tried your string 200 times w/o success and it works every time in the browser...Don't know what I could be doing wrong...

      I don't think it is the lyrics web site as the first ->get() works, it is just the 2nd one that fails...

      The link that failes in www:mechanize is:

      http://display.lyrics.astraweb.com:2000/display.cgi?beatles..beatles_1..hey_jude

      This URL works in my browser always...

      Here is my complete test program incase someone else wants to give it a try ;-)

      use WWW::Mechanize; use URI::URL; use strict; use warnings; my $artist = 'The Beatles'; my $title = 'Hey Jude'; my $mech = WWW::Mechanize->new(autocheck=>1); #$mech->agent_alias('Windows IE 6'); my $search = join("+", split(/ /, $artist)) . "+" . join("+", split(/ +/, $title)); print "search=" . "http://search.lyrics.astraweb.com/?word=$search" . + "\n"; $mech->get("http://search.lyrics.astraweb.com/?word=$search"); $mech->success() or die "Can't get the search page\n"; $mech->follow_link(text=>"Hey Jude"); #$mech->follow_link(n=>6); $mech->success() or die "Can't find lyrics page\n";
        Your script works for me as often as following the like works for me in a browser.

        I'd suggest that you try to get some more information about what's going on. Instead of just die-ing after a failure, you might try printing out $mech->status and/or $mech->content to see if that sheds any light on what's going on.

        Sorry I can't be of more help.

        Linda

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://332809]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (12)
As of 2014-09-02 16:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite cookbook is:










    Results (25 votes), past polls