Strip HTML tags again

[dda]

Hello All!

Can you give me a simple example of how to filter out all HTML tags in a single line of text, but 'non-HTML' tags should not be filtered? For example:

Should be filtered, the result will be "text1":

<a href="mylink>text1</a>

Should not be filtered, the result will be "<this is a normal text>":

<this is a normal text>

Your help is greatly appreciated.
--dda

[Ovid]

This problem looks tailor-made for my HTML::TokeParser::Simple module, when combined with HTML::Tagset. The following test will demonstrate:

#!/usr/bin/perl -w
use strict;
use HTML::TokeParser::Simple;
use HTML::Tagset;

my $html = <<'END_HTML';
<a href="mylink">text1</a>
<this is normal text>
END_HTML

my $p = HTML::TokeParser::Simple->new( \$html );

while ( my $token = $p->get_token ) {
    next if ! $token->is_text 
              and 
              exists $HTML::Tagset::isKnown{ $token->return_tag };
    print $token->return_text;
}

Result:

text1
<this is normal text>

Cheers,
Ovid

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

[Your Mother]

++ for the original. I'm posting an updated example because some changes to the module seem to have borked your example. This is an in place stripper--based on the one you posted--with the newer/working syntax.

sub strip_html {
    my $renew = "";
    my $p = HTML::TokeParser::Simple->new(\$_[0]);
    no warnings "uninitialized";
    while ( my $token = $p->get_token ) {
        next if ! $token->is_text
            and
            exists $HTML::Tagset::isKnown{ $token->get_tag };
        $renew .= $token->as_is;
    }
    $_[0] = $renew;
}

[merlyn]

Here's an example from the eg directory in the HTML::Parser distribution:

#!/usr/bin/perl -w

# Extract all plain text from an HTML file

use strict;
use HTML::Parser 3.00 ();

my %inside;

sub tag
{
   my($tag, $num) = @_;
   $inside{$tag} += $num;
   print " ";  # not for all tags
}

sub text
{
    return if $inside{script} || $inside{style};
    print $_[0];
}

HTML::Parser->new(api_version => 3,
                  handlers    => [start => [\&tag, "tagname, '+1'"],
                                  end   => [\&tag, "tagname, '-1'"],
                                  text  => [\&text, "dtext"],
                                 ],
                  marked_sections => 1,
        )->parse_file(shift) || die "Can't open file: $!\n";;

-- Randal L. Schwartz, Perl hacker

[cjf]

Have a look at HTML::Tagset it contains various lists of valid HTML tags for different sections of a document.

Update: ++ to Ovid for providing the working example below.

[dda]

Thanks!!! It is the stuff I was looking for. Now I'd like to know how to use it in a 'perl' manner. Currently I have the following code (right from perlfaq):

sub strip_html {
    my $t = shift;
    $t =~ s/<(?:[^>'"]*|(['"]).*?\1)*>//gs;
    return $t;
}

Seems like I have to use %HTML::Tagset::isKnown hash, but how to apply it to my sub? I can't find any quick way...

--dda

[tachyon]

Re: Tag filtering: a standard mechanism?

cheers

tachyon

s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[thpfft]

heh. i ended up writing HTML::TagFilter because tachyon shouted at me so loud. Which only does part of what you want, sadly, so I wouldn't recommend it. But i'm usefully reminded to finish the next version, which does the rest. And lots of other exciting things, i feel sure.

[ides]

This will probably do the trick, however this does not handle HTML tags that span multiple lines. To do that you'll most likely have to join all the lines together into one scalar. This will also not catch multiple HTML tags on the same line, you'll need to modify it to suit your needs.

What this is doing is finding text contained in <>'s that has a corresponding ending tag.

Here is the code ($l is the scalar holding the line of text):

if( $l =~ /<.*?>(.*?)<\/.*?>/ ) { 
    $l = $1; 
}

-----------------------------------
Frank Wiles <[e-mail]>
http://frank.wiles.org

[dda]

Thanks, but I need a solution which 'knows' about possible HTML tags. What I need is to filter HTML from a chat message, and if someone type '<Hehe>' - it will be wiped off.

--dda

[hacker]

Have you looked at HTML::LinkExtor? It sounds to be exactly what you want:

HTML::LinkExtor is an HTML parser that extracts links from an HTML document. The HTML::LinkExtor is a subclass of HTML::Parser. This means that the document should be given to the parser by calling the $p->parse() or $p->parse_file() methods.

I've used it successfully in the past with a lot of my parsing code.

[Mask]

Hi monks, i am little bit disappointed in all this discussion.
If the input from chat is displayed in HTML page, then any "<" or ">" in the displayed text will be transformed to the &lt; and &gt; . So if you can see <this is a normal text> in your web browser, than in the sources of a HTML page it will be &lt;this is a normal text&gt; in this case you should not be bothered about knowing all tags, and if you want to see the text as it is in browser you need just to replace "&lt;" by "<" and "&gt;" by ">" in your perl code.

[dda]

Ok, I'll try to explain. If someone types '<b>some text</b>', it should not be be displayed as a bold text in chat window, and displaying the HTML source is not a good idea too. I just have to strip all tags from the line.

And '<some text>' should not be stripped. All regexp-based solutions will strip this too. I'm still looking for regexp-based stuff that will use %HTML::Tagset::isKnown hash to filter out only correct HTML tags.

--dda

[little]

look up the POD (or your preferred docs) for HTML::Tagset
cite: "hashset %HTML::Tagset::isKnown
This hashset lists all known HTML elements."
So you've got to compare your match with that list ...

Have a nice day
All decision is left to your taste

Addendum

Look through the previous suggestions as well. Try it at least and ask again if you get an error or get otherwise stuck. :-)

[mousey]

$foo =~ s/<(.|\n)+?>//g;


This is great! from up here we can throw lots and lots of stuf! but uh...how do we get down? --Goblin Balloon Brigade