Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses
 
PerlMonks  

HTML::TokeParser::Smart

by lshatzer (Friar)
on Apr 09, 2002 at 00:53 UTC ( #157614=sourcecode: print w/ replies, xml ) Need Help??

Category: Web Stuff
Author/Contact Info lshatzer
Description: This will detect if it is a URL, file, or html and pass it to HTML::TokeParser, and returns the HTML::TokeParser object. (This was my first venture into inheritance.)
Updated: Changed a few things from Amoe's suggestions.
package HTML::TokeParser::Smart;
require 5.006;
use strict;
use warnings;

use Carp;
use LWP;
use base 'HTML::TokeParser';
our $VERSION = '0.2';

sub new {
  my $proto = shift;
  my $class = ref($proto) || $proto;
  my $url = shift;
  my $self;
  if (-e $url) {
    # It's a file!
    $self = HTML::TokeParser->new($url);
  }
  elsif ($url =~ m/^https?|^ftp|^file/) {
    # It's a URL!
    my $browser = LWP::UserAgent->new;
    my $req = $browser->request(HTTP::Request->new(GET=>$url));
    croak "Unable to get webpage: $url ", $req->status_line unless $re
+q->is_success;
    $self = HTML::TokeParser->new($req->content_ref);
  }
  elsif ($url =~ m/<[^>]+>/) {
    # It's HTML!
    $self = HTML::TokeParser->new(\$url);
  }

  else {
    croak "'$url' is neither a valid URL, file, or HTML.";
  }
  bless ($self, $class);
  return $self;
}

1;

Comment on HTML::TokeParser::Smart
Download Code
Re: HTML::TokeParser::Smart
by Amoe (Friar) on Apr 09, 2002 at 10:22 UTC

    Aww, I was hoping it would automagically figure out which bits of HTML I wanted... ;)

    Couple of points to make about this, though:

    • Saying use base 'HTML::TokeParser' is IMHO a lot cleaner than that BEGIN mess.
    • If your class user provides a URL with angle brackets in and forgets to URI-escape them, you'll pass it as HTML, which probably won't dwim. Perhaps there's a module you could use to check if a file's HTML?
    • Using LWP::Simple makes saying that middle bit a lot easier, and since you're not using any of the extra featues of UserAgent its use is redundant here.

    Actually, you could solve that middle problem quite neatly by just switching the order of the tests so that it went (file - url - html) rather than (file - html - url), since all URLs acceptable by LWP will have a protocol, and URIs in HTML must be wrapped in a tag.

    --
    amoe
      Thanks for your input, I also recived some suggestons from crazyinsomiac to take a look at URI::URL, since I knew my url checking was not the best, and I had planned on changing it.

      Your last bullet point, see my use.perl journal, it has some info from Sean Burke. Any other suggestions would be welcome.

      Also, a better name would be good, another comment from crazyinsomniac says the Smart part dosn't indicate what the module does, but does not have any suggestions...

      Some other ideas I had was HTML::TokeParser::URL, but it will also do files or html.

      On an unrelated note, this is my 42nd post, not a 'major' milestone, but I think it's cool.
        lshatzer, I am very interested to use this package since I desparately need to URL in conjunction with HTML::TokeParser.
        I am new to LWP stuff.
        Would you please give some pointer as to how I can use this package? Thank you

Back to Code Catacombs

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://157614]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others avoiding work at the Monastery: (9)
As of 2014-07-31 11:47 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (248 votes), past polls