|Just another Perl shrine|
Parsing SGML-ish Data Filesby coolmichael (Deacon)
|on Aug 01, 2012 at 17:10 UTC||Need Help??|
coolmichael has asked for the
wisdom of the Perl Monks concerning the following question:
Hello Monks, it's been a while (only seven years!).
I'm working on a project to parse, validate, and transform some large data files which look like SGML tags, but most definitely are not. For example, <FOO=4> is perfectly valid and well defined, and the tags do not have to nest properly as they do in SGML and XML. I've tried the SGML:: tools on CPAN, but they don't quite work. I've also tried HTML::Parser, but it chokes on attributes which use "smart quotes" (0x201D in Unicode).
I've written a pure perl finite state machine parser (and test suite) which creates a data structure I can validate, but it is very slow. Like 45 seconds on a 900Kb file. The bottleneck is the parsing phase, so I'd like to speed that up somehow.
I've squeezed as much performance out of it as I can with Devel::NYTProf, but I think if I want to get it down to 10 seconds a file I need to rewrite the parser some how. I could go the C/XS route for it, but that would be a massive learning curve.
I haven't tried Parse::RecDescent yet or Parse::Yapp. What are your thoughts on them?
If you were writing a parser for something SGMLish (but not SGML), where would you start?