Gathering experiences with Parse::RecDescent

[leriksen]

I'm presenting a talk on my experiences of Parse::RecDescent for the Open Source Developers Conference 2005. But I think it would be interesting to gather a wider range of experience, so I'm asking for other monks experiences and opinions on the module.

It does have a fearsome reputation, mainly ( I feel) due to the complex area it addresses.

How was your first experience ?
Did you go on to use it for other projects ?
Did you attain mastery ?
Are you scared to ever go back ?

Tips, triumphs and tragedies are all welcome!

[Corion]

I found Parse::RecDescent nice, as I am fond of recursive descent parsers - the top-down approach of parsing seems elegant to me and it can produce nice, sensible error messages. The surrounding utility functions to automatically generate ASTs etc. are also nice.

That said, I found two showstopping drawbacks to Parse::RecDescent. First of all, it's horribly slow for moderately complex grammars (Javascript in my case). Second, if you have a syntax error in your grammar, the error messages range from obscure to opaque, and you can't go and fix the parser itself, because Parse::RecDescent is written in itself and the distribution (at least at the time when I used it) didn't include the source grammar but only the generated result. From time to time, it wouldn't raise an error at all, or raise an error in the wrong place.

Personally, I recommend Parse::YAPP nowadays for any serious parsing (if it has to be done in Perl). The drawback of Parse::YAPP is that it needs a left-recursive grammar where Parse::RecDescent wants a right-recursive grammar (or vice versa), but the resulting code is nice, the driver for the parser is simple and could be written in XS to get a moderate speed gain.

[jmcnamara]

I used Parse::RecDescent mainly for the formula parser in Spreadsheet::WriteExcel and I found it relatively easy to learn and use.

The learning phase was greatly helped by the 40 demo programs and the tutorial. The usage phase was helped by the combination of lexer and parser in one.

However, it is very slow. The (vaguely) promised Parse::FastDescent never materialised and probably never will given the enhanced parsing features in Perl6 regexes.

In the end I moved/am moving to Parse::Yapp since it is faster and having a Yacc-like grammar is useful for porting to other languages.

If you are looking for other real-life experiences with P::RD then this article gives some practical observations in relation to using Parse::RecDescent, Parse::Yapp, Perl-byacc and regular expressions in a real project.

--
John.

[xdg]

I was definitely intimidated by it at first. The chapter on parsing and the examples using Parse::RecDescent in Advanced Perl Programming, 2nd ed. made it much less so and I decided to give it a try on my recent Pod::WikiDoc module for the wiki-text parsing part of it. It went fairly well and was almost fun to figure out -- that said, it's very hard to debug and follow what it's doing when it isn't going well.

I'd consider using it again, but now that I know more about parsing (thanks in part to Higher Order Perl), I might consider other techniques/modules as well.

[philcrow]

I used Parse::RecDescent in Data::HTMLDumper to give control over how the html tables actually look. I'm also using it on a new project which attempts to describe a web application in a single file, which I parse with Parse::RecDescent so various backend modules can generate things like controllers, Class::DBI subclasses, sql to build the database, etc.

Since I am self taught, I find the recursive descent style easier to understand, explain, and fix than the LALR style (with its shift/shift and shift/reduce conflicts). Using the trace option usually makes it easy to find where the parse went south.

I do notice the speed, but it is fast enough even for the moderately complex task of parsing my web app definition files (the current app has a 400 line definition). But this is a development tool. I can certainly wait 25-45 seconds for my the parser and backends to produce the basics of my whole app.

I'm also glad to see that grammars are coming to Perl 6. That may give a speed benefit, but it should also help vi with color highlighting.

Phil

[amw1]

I used P:RD about a year ago to parse an intermediate 'language' that was generated by a web. I parsed the language into a datastructure so I could re-represent the clauses the user entered when they went back to edit the block they created. We then took the intermediate language and used lisp to parse and execute the appropriate stuff. It took me a few weeks to get to the point where I was comfortable with it, but overall I don't remember any real pain. (I had never written a true parser before so there was a bunch of learning that I needed to do during that time)

I haven't had the opportunity to use it again but there are a number of other people in my office who have used it to do various tasks. I have been able to support them reasonably well. With only one project using it, I doubt I'd say I attained mastery, but I am confident that I can use it again with little effort if the need should arise.

I am far from scared of going back. I've been wanting another project that needs to use P:RD. I found that I really enjoyed writing the grammar.

One tip I would offer, if speed matters stay away from the lookahead stuff. (In my experience the lookaheads seemed make the parsing crawl. If I were better at language design I could have probabbly avoided the issue alltogether.)

[samtregar]

I've used Parse::RecDescent a few times, most recently in creating HTML::Template::Expr. It's a fun module and one I recommend all Perl programmers learn, if only for the experience. It's a fine example of the extent to which mini-languages embeded in Perl can make hard problems easier to solve.

However, it is worth noting that the module is horribly slow, even running a carefully tuned grammar. Damian had plans to address this but it seems they never came to fruition. I've been forced to avoid its use in a number of projects because the speed penalty was just too high.

-sam

[TStanley]

I used it in creating my Config::Yacp module. I did a lot of reading of the documents at first, then took the plunge into it. I did find it easy to use once I understood how it worked.

Eventually, I need to take a look at the source code again and figure out how I can improve it.

[drbean]

I tried it after working with m/\G/gc regex first. I was surprised how easy it was converting my regex to a Parse::RecDescent grammar. I have begun to see where the talk in perl6 about rules rather than regex is coming from. I like the feeling in perl of being able to pull yourself up by your bootstraps, learning as you go. At the same time, I wonder why whatever I read in the perl5 stuff never tried to lead me across the bridge from regex. Now I'm reading Dick Grune's online book about parsing. I've only written one simple grammar. The feeling of mastery however is motivating. I think I will be on the lookout for ways to parse more things from now on. All this was generated by my first simple parsing need, however, of course.

[polypompholyx]

I've used it to write toy calculator grammars, a parser for chemical formulae that calculates molecular masses, and to convert a user-friendly multiple-choice question format into Something Nasty required by an e-learning system (WebCT). It's simple to use, but like others, I found it slow (although precompiling the parser helps). On the other hand, I often find myself thinking, "Oh, I could do that much more elegantly in P::RD," when presented with a problem that I would previously have used nasty regexes on.

[leriksen]

Ah yes - webCT - also known as "Run Away, RUN AWAY!!!!"

[Anonymous Monk]

For what it's worth, I documented my first experience with P::RD in this thread at the DevShed forum: DevShed

I did go on to use the module again, but usually fall back on simpler methods to extract text.