Re^2: Perl Monks hypocrisy (HTML parsing speed)

[tye]

But a set of regexes that parses any HTML document correctly is much less efficient than something based on HTML::Parser.

As I recall, we tried a module based on HTML::Parser but had to drop it because it was way too slow (10-times slower, IIRC). PM uses a single regex to split the HTML into tokens and another regex to deal with filtering attributes in those tokens.

There are two main reasons that I'd advise someone to not "parse HTML with (a) regex(es)". Performance is not one of them.

The main point is that you probably shouldn't use something like /<td>(.*?)</td>/ because there is no way to make that ignore HTML comments that contain similar HTML. The other is that doing such can look easy but end up being very hard so it is often less work in the long-run to use a decent module from the start, even though that often looks like a more difficult approach.

Update: The "HTML" that we parse is stuff typed in by our users "by hand". So our HTML parser (the regex) intentionally deals with certain border cases in specific ways. No, it does not strictly follow any one of the many HTML standards we have to choose from.

                - tye

[Juerd]

As I recall, we tried a module based on HTML::Parser but had to drop it because it was way too slow (10-times slower, IIRC).

The speed has everything to do with the complexity of your parser. If you don't need to follow specifics, and don't need to implement the usual browser quirks, a single regex is often a lot more efficient. It's up to the end user to benchmark it. Unfortunately, most novices don't know how to write the regex, don't know how to write an HTML::Parser based scripts and don't know how to benchmark.

Juerd # { site => 'juerd.nl', plp_site => 'plp.juerd.nl', do_not_use => 'spamtrap' }