Debugging regular expressions

[larsen]

Today I had to write a regular expression definitely more complex than I'm used to do (I'm a real inept: for example this is the first time I use x modifier in REs :)).

This experience showed me I spent a lot of time trying to make my RE working, specially (in my opinion) 'cause I'm not able to debug it in a good way.

I used some tecniques in order to make my work faster (well, I should say "not so slow"...):

• x modifier. I think the first debug trick is always avoid debug, i.e. write correct code :). And to write correct RE a good trick is comprehend what are you writing. Whitespaces and comments are very good and, last but not least, they let you...
• proceed incrementally. I wrote my code incrementally, commenting large chunks of the regular expression and debugging only small pieces

I guess there are other tricks of ways to code better and faster (faster! faster than before... :)), like:

• prepare case tests instead of using only actual data that will be processed by the RE.
• exploit the isomorphism between RE and state machines. Did you ever simulated by hand your RE (I mean, a "real" RE) using paper, pencil, coins and drawing a state machine? (and using a computer?
• I'd really like to discover there's a way to follow what's happening at "core level" when Perl try to satisfy a RE. This is not very good from my point of view (I think this kind of "insight" perturbs what's happening at core level), but I also think that such a possibility should save me some time this afternoon. Maybe something like this?

What do you think about?

[wog]

* I'd really like to discover there's a way to follow what's happening at "core level" when Perl try to satisfy a RE.

use re 'debug';

... The re pragma can give you a hint (and possibly help you understand why you regexp doesn't do what you think it does). What it does output is not necessairly easy to understand.

update: To address what bunsunsl said: this feature has been avialable since perl 5.005 and tests reveal it does not require the -DDEBUGGING to have been used when perl was compiled.

[Ovid]

You'll probably want to read this node about testing regular expressions and the replies to it. Very helpful.

Cheers,
Ovid

Join the Perlmonks Setiathome Group or just click on the the link and check out our stats.

[Abigail]

It also helps to "translate" the specification of how the to be parsed data looks like into a language that easily maps to regular expressions. As in:

3 or 5 digits, a dash, 4 to 6 uppercase characters, a slash or whitespace, 2 digits and the letter K

Once you have nailed down the specification, the implementation is often quite easy.

-- Abigail

[Albannach]

I don't know why nobody has mentioned it yet, but I find YAPE::Regex::Explain (written by our own japhy) great for experimenting with different regex recipes, and certainly for deciphering what the masters have written, which speeds my learning process.

--
I'd like to be able to assign to an luser

[clemburg]

I think the first debug trick is always avoid debug, i.e. write correct code :). And to write correct RE a good trick is comprehend what are you writing.

It also helps to "translate" the specification of how the to be parsed data looks like into a language that easily maps to regular expressions.

A good technique to avoid errors and to code a regex according to specification is to build up the regex from strings stored in variables, which you then interpolate into the regex. This also makes the regex much more readable for others. Re: Trying to avoid line noise (brain cramp) gives an example.

Christian Lemburg
Brainbench MVP for Perl
http://www.brainbench.com

[busunsl]

With Perl 5.6 you can use 'use re "debug"' and will get something like a trace of the regex.
It's not easy to read, but it helps.
I think you must have perl compiled with the DEBUGGING flag.

Look at perldoc perldebguts and search for 'regular'.