When Perl was first released, regular expressions matched bytes. That means you could forget about Unicode. Later, Unicode support was added so you could properly match characters, but after reading Dominus' article about how to build a regular expression engine, it occurred to me that instead of matching bytes or characters, I could match tokens.

What follows is a very simple demonstration of the Token::Regex module that you can download from this link. I'm not putting it on CPAN (but perhaps in the future) because it's very alpha code and, in any event, it's just a hack on the Regex.pm module that Dominus provided, so I'm not sure that my uploading would be appropriate prior to asking for his consent (of course, this code is so fragile I'd be embarrassed to upload it).

First, I need to create appropriate tokens. In my scheme, tokens merely need to have an identifier() method to work. So lets create a small class that allows HTML tokens that are tags to produce identical identifiers if and only if they are the same type of token and have identical attribute names (regardless of order or value). In other words, the following tokens will be considered identical for the purpose of a regular expression match:

<p class="foo" name="bar">
<p name="bar" CLASS="ovid has no class">
<p NAME="bar" class="ovid has no class">
<p name="bar" class="ovid has no class">
[download]

Here's one way we could code the package:

package HTML::Token;

sub new {
    my ($class,$token) = @_;
    my $self = bless { token => $token }, $class;
    my $identifier = $self->identifier;
    return $self;
}

sub identifier {
    my $self  = shift;
    my $token = $self->{token};
    my $attributes = $token->return_attrseq;
    my $tag = $token->return_tag;
    if (ref $attributes eq 'ARRAY') {
        return sprintf "%s %s", $tag, join ' ', sort @$attributes;
    }
    else {
        return $tag;
    }
}
[download]

Now let's create a regular expression array. To use this, every element is either a token or a single regex meta character. See the POD for an allowed list of meta characters.

my $html = <<END_HTML;
<h1>This is a test</h1>
<p class="foo" name="bar">so what???
END_HTML
my $parser = HTML::TokeParser::Simple->new(\$html);
my @tokens = ();
while (my $token = $parser->get_tag) {
    push @tokens => HTML::Token->new($token);
}
push @tokens => (qw[* . *]); # make the (p) tag zero or more, followed
+ by anything
[download]

Note the last line. With this regex engine, all regexes are bound to the beginning of the array and the end of the array. To "unbind" them, you have to include a dot star at the beginning and end.

The above regular expression says "match an opening H1 tag with no attributes followed by a closing H1 tag followed by one or more P tags with attributes of "class" and "name" followed by zero or more of anything". (deep breath).

And to set it up:

use Token::Regex;
my $regex = Token::Regex->new('HTML::Token');
$regex->parse(\@tokens);
[download]

We're now ready to match some HTML. Just provide an array ref of tokens to the match() method:

my $tokens = html_tokens(<<END_HTML);
<h1>This is html</h1>
<p name="bar" CLASS="ovid has no class">so what???
<p NAME="bar" class="ovid has no class">so what???
<p name="bar" class="ovid has no class">so what???
<h2>and this is okay</h2>
END_HTML

if ($regex->match($tokens)) {
    print "Yes\n";
}
else {
    print "No\n";
}

$tokens = html_tokens(<<END_HTML);
<p name="bar" CLASS="ovid has no class">so what???
<p NAME="bar" class="ovid has no class">so what???
<p name="bar" class="ovid has no class">so what???
<h2>and this is okay</h2>
END_HTML

if ($regex->match($tokens)) {
    print "Yes\n";
}
else {
    print "No\n";
}


sub html_tokens {
    my $html = shift;
    
    my $parser = HTML::TokeParser::Simple->new(\$html);
    my @tokens;
    while (my $token = $parser->get_tag) {
        push @tokens => HTML::Token->new($token);
    }
    return \@tokens;
}
[download]

The first bit of HTML should match, but the second should fail (due to no H1 tag).

As stated previously, this is very fragile code, so using this in production is not a good idea. However, note that it can be used with any tokens that you care to create so long as you create an appropriate identifier() method. Further, backreferences cannot be added to this code and I don't know enough about regular expression engines to rewrite the code to support that. Parentheses are currently for grouping.

Furthermore, the code is difficult to use and ugly as sin. It's also slow and I don't know that anyone would really have a use for it, but I thought it was a neat hack. Any suggestions for making it easier would be welcome.

Cheers,
Ovid

New address of my CGI Course.