Parsing HTML with tags intact

mr_p has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

I am trying to parse html text with tags intact. The code I wrote using HTML::Parser strips out the tags.

Is there a way to keep tags intact?

below is my code.

#!/usr/bin/perl
package MyParser;
use base qw(HTML::Parser);
my $main_content="";
sub start {
      my ($self, $tag, $attr, $attrseq, $origtext) = @_;

        if ($tag =~ /^span$/i && $attr->{'class'} =~ /^main-content$/i
+)
        {
            # set if we find <span class="main-content"
            $content_flag = 1;
        }
  }
sub text {
        my ($self, $text) = @_;
        # If we're in <H1>...</H1> or
        if ($content_flag)    { $main_content    .= $text; }
}
my $html = " <html>
    <head>
        <title>Blah</title>
    </head>
    <span class=\"main-content\">
    <bold_text> Here's the body 1 </bold_text>
    <p> para1 </p>
    <p> para2 </p>
    </span>
    </html>";

my $parser = MyParser->new;
$parser->parse("$html");
print "$main_content\n";
[download]

Output I get:

         Here's the body 1
         para1
         para2
[download]

Output I need is:

    <bold_text> Here's the body 1 </bold_text>
    <p> para1 </p>
    <p> para2 </p>
[download]

I would like the above output to still have tags, is it possible to do this with HTML::Parser?

Comment on Parsing HTML with tags intact Select or Download Code

Replies are listed 'Best First'.
Re: Parsing HTML with tags intact by roboticus (Chancellor) on Jan 06, 2011 at 23:52 UTC
mr_p: The first example in the docs gives a good hint on how to do it. ...roboticus When your only tool is a hammer, all problems look like your thumb.	[reply]
Re: Parsing HTML with tags intact by Anonyrnous Monk (Hermit) on Jan 06, 2011 at 23:51 UTC
Kind of hackish... but as the `start` handler is being called before the `text` handler, you could store the tag name in a global variable, which you can then use in the `text` method to decorate the content as desired: `... my $tagname; sub start { my ($self, $tag, $attr, $attrseq, $origtext) = @_; $tagname = $tag; # store tag name for later use if ($tag =~ /^span$/i && $attr->{'class'} =~ /^main-content$/i +) { # set if we find <span class="main-content" $content_flag = 1; } } sub text { my ($self, $text) = @_; # If we're in <H1>...</H1> or my $tagged_text = $text !~ /^\s$/ ? "<$tagname>$text</$tagnam +e>" : $text; if ($content_flag) { $main_content .= $tagged_text; } } ...` [download] Output: `<bold_text> Here's the body 1 </bold_text> <p> para1 </p> <p> para2 </p>` [download] (Might need some further tweaking (like the `$text !~ /^\s$/ ? ...`) to handle edge cases — but you get the idea.)	[reply] [d/l] [select]
Re: Parsing HTML with tags intact by ww (Archbishop) on Jan 07, 2011 at 03:28 UTC
Your "html" is just a touch odd. It contains two obvious departures from valid html. The data provided lacks a `<body>...<body>` tag pair and `<bold_text>` isn't an html tag at all. Is what you posted merely a misguided, fanciful illustration or are you actually confronted with such data?	[reply] [d/l] [select]
Re^2: Parsing HTML with tags intact by mr_p (Scribe) on Jan 07, 2011 at 13:51 UTC
I can have misguided data as such. I have to be able to handle such data.	[reply]
Re^3: Parsing HTML with tags intact by ww (Archbishop) on Jan 07, 2011 at 14:17 UTC
In that case, `perldoc -f HTML::Parser` raises the question (for me, anyway): "Is H::P the appropriate tool? "HTML::Parser" is not a generic SGML parser. We have tried to make it able to deal with the HTML that is actually "out there", and it normally parses as closely as possible to the way the popular web browsers do it instead of strictly following one of the many HTML specifications from W3C. Where there is disagreement, there is often an option that you can enable to get the official behaviour." But I have yet to see a browser that will treat `<bold_text>` as a tag, rather than simply ignoring it (and merely rendering the enclosed text without additional formatting. Update: The doc does -- again, to me and YMMV -- suggest that you could tweak your script to deal with some of the invalid tag-like entries: $p->strict_names $p->strict_names( $bool ) By default, almost anything is allowed in tag and attribute names. This is the behaviour of most popular browsers and allows us to parse some broken tags with invalid attribute values like: <IMG SRC=newprevlstGr.gif ALT=PREV LIST BORDER=0> By default, "LIST]" is parsed as a boolean attribute, not as part of the ALT value as was clearly intended. This is also what Mozilla sees. The official behaviour is enabled by enabling this attribute. If enabled, it will cause the tag above to be reported as text since "LIST]" is not a legal attribute name. "What may be possible" and "what's provided by this node" are -- however -- two different things. :-\	[reply] [d/l] [select]


Come for the quick hacks, stay for the epiphanies.
	PerlMonks