HTML::TokeParser::Smart

[lshatzer]

Category:

Web Stuff

Description:

This will detect if it is a URL, file, or html and pass it to HTML::TokeParser, and returns the HTML::TokeParser object. (This was my first venture into inheritance.)
Updated: Changed a few things from Amoe's suggestions.

package HTML::TokeParser::Smart;
require 5.006;
use strict;
use warnings;

use Carp;
use LWP;
use base 'HTML::TokeParser';
our $VERSION = '0.2';

sub new {
  my $proto = shift;
  my $class = ref($proto) || $proto;
  my $url = shift;
  my $self;
  if (-e $url) {
    # It's a file!
    $self = HTML::TokeParser->new($url);
  }
  elsif ($url =~ m/^https?|^ftp|^file/) {
    # It's a URL!
    my $browser = LWP::UserAgent->new;
    my $req = $browser->request(HTTP::Request->new(GET=>$url));
    croak "Unable to get webpage: $url ", $req->status_line unless $re
+q->is_success;
    $self = HTML::TokeParser->new($req->content_ref);
  }
  elsif ($url =~ m/<[^>]+>/) {
    # It's HTML!
    $self = HTML::TokeParser->new(\$url);
  }

  else {
    croak "'$url' is neither a valid URL, file, or HTML.";
  }
  bless ($self, $class);
  return $self;
}

1;

[Amoe]

Aww, I was hoping it would automagically figure out which bits of HTML I wanted... ;)

Couple of points to make about this, though:

• Saying use base 'HTML::TokeParser' is IMHO a lot cleaner than that BEGIN mess.
• If your class user provides a URL with angle brackets in and forgets to URI-escape them, you'll pass it as HTML, which probably won't dwim. Perhaps there's a module you could use to check if a file's HTML?
• Using LWP::Simple makes saying that middle bit a lot easier, and since you're not using any of the extra featues of UserAgent its use is redundant here.

Actually, you could solve that middle problem quite neatly by just switching the order of the tests so that it went (file - url - html) rather than (file - html - url), since all URLs acceptable by LWP will have a protocol, and URIs in HTML must be wrapped in a tag.

--
amoe

[lshatzer]

Thanks for your input, I also recived some suggestons from crazyinsomiac to take a look at URI::URL, since I knew my url checking was not the best, and I had planned on changing it.

Your last bullet point, see my use.perl journal, it has some info from Sean Burke. Any other suggestions would be welcome.

Also, a better name would be good, another comment from crazyinsomniac says the Smart part dosn't indicate what the module does, but does not have any suggestions...

Some other ideas I had was HTML::TokeParser::URL, but it will also do files or html.

On an unrelated note, this is my 42nd post, not a 'major' milestone, but I think it's cool.

[Anonymous Monk]

lshatzer, I am very interested to use this package since I desparately need to URL in conjunction with HTML::TokeParser.
I am new to LWP stuff.
Would you please give some pointer as to how I can use this package? Thank you