http://www.perlmonks.org?node_id=146275

nop has asked for the wisdom of the Perl Monks concerning the following question:

I have a simple program to check the validity of links before submitting them to search engines (code below.)

This code declares that "incomplete" URLS, like  http://www.mysite.com/mydir are no good -- when, in fact, they work just fine in a browser (and when submited to search engines). The "problem" with these URLs is that the full path all the way down to file isn't explicitly specified, I think.

My question is how do I get LWP useragent to act like a browser and find the default page in a directory?

thanks

nop

package MyUA; use base qw(LWP::UserAgent); use strict; use CGI qw/:standard/; sub redirect_ok {1}; sub new { my $class = shift; my $self = new LWP::UserAgent; bless($self, $class); return $self; } sub validURL { my ($self, $url) = @_; my $req = new HTTP::Request POST => $url; my $res = $self->request($req); my $content = $res->content; return 0 unless $res->is_success; return 0 if $content =~ /the page you have requested cannot be fou +nd/i; return 1; } 1;