comment on

I have a simple program to check the validity of links before submitting them to search engines (code below.)

This code declares that "incomplete" URLS, like http://www.mysite.com/mydir are no good -- when, in fact, they work just fine in a browser (and when submited to search engines). The "problem" with these URLs is that the full path all the way down to file isn't explicitly specified, I think.

My question is how do I get LWP useragent to act like a browser and find the default page in a directory?

thanks

nop

package MyUA;
use base qw(LWP::UserAgent);
use strict;

use CGI qw/:standard/;


sub redirect_ok {1};

sub new {
    my $class = shift;
    my $self = new LWP::UserAgent;
    bless($self, $class);
    return $self;
}

sub validURL {
    my ($self, $url) = @_;
    my $req = new HTTP::Request POST => $url;
    my $res = $self->request($req);    
    my $content = $res->content;
    return 0 unless $res->is_success;
    return 0 if $content =~ /the page you have requested cannot be fou
+nd/i;
    return 1;
}

1;
[download]

In reply to Checking "incomplete" URLs by nop

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Do you know where your variables are?
	PerlMonks