http://www.perlmonks.org?node_id=305173

svsingh has asked for the wisdom of the Perl Monks concerning the following question:

I'm trying to pull some statistics off a Yahoo page using LWP::Simple's get method. I noticed that my script would sometimes seem to hang inconsistently, so I stripped down the program to see if I could identify the problem.

I think get is hanging somewhere, but I don't understand why. I read a few other posts here, but wasn't able to find any advice on how to work around the problem. I don't think it's a timeout issue, because the program isn't running the "or die..." code.

Here's the stripped down script:

use strict; use LWP::Simple; my $leagueURL = 'http://sports.yahoo.com/nhl/stats/byposition?pos=C,RW +,LW,D'; my $data = get $leagueURL || die "Couldn't get $leagueURL"; while ( $data =~ m|<a href="/nhl/players/(\d+)">|gs ) { my $id = $1; my @stats = ($id); print STDERR "Getting player $id\n"; push( @stats, &getCareer($id) ); push( @stats, &getSplits($id) ); print STDOUT "@stats\n"; sleep(15); } sub getCareer { my ($id) = @_; my $url = "http://sports.yahoo.com/nhl/players/$id/career"; my $data = get $url || die "Couldn't get $url"; return ("gotCareer"); } sub getSplits { my ($id) = @_; my @s = (); my $url = "http://sports.yahoo.com/nhl/players/$id/splits?year=care +er"; my $data = get $url || die "Couldn't get $url"; return ("gotSplits"); }

I've run the stripped code four times now. The first time, it went through the while loop 7 times. Then 4, 2, and 6. In the current run, it's been working on the same iteration for the past twenty minutes.

I would expect if the request was timing out, that I'd be seeing the "Couldn't get $url." message, but that's no happening. Am I doing something wrong? Thank you.

Update: Changed $leagueURL to refer to real page instead of local file.