Beefy Boxes and Bandwidth Generously Provided by pair Networks DiBona
Just another Perl shrine
 
PerlMonks  

Testing regex equivalence

by loris (Hermit)
on Feb 21, 2006 at 11:48 UTC ( #531661=perlquestion: print w/ replies, xml ) Need Help??
loris has asked for the wisdom of the Perl Monks concerning the following question:

Monks,

What would the best way of testing whether two regexes are equivalent? In particular, I am interested in seeing whether some apparently complex regex is actually equivalent to

.*

I could obviously construct a set of strings which are somehow typical and see whether both regexes selected the same set, but that would demonstrate only rather limited equivalence and seems a bit of lame approach.

Thanks,

loris


"It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ."

Comment on Testing regex equivalence
Download Code
Re: Testing regex equivalence
by mickeyn (Priest) on Feb 21, 2006 at 12:45 UTC
    can you post the regex you wish to check ?
    Mickey

      I don't want to check a specific regex. I want to check a large number of them.

      Thanks,

      loris


      "It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ."
Re: Testing regex equivalence
by Corion (Pope) on Feb 21, 2006 at 12:57 UTC

    I'm not sure what exactly your goal is. Every regular expression of a certain size either has no loops or it has loops, and if it has loops, there is a certain string length after which parts of the regular expression repeat (I believe this is more or less the pumping lemma). So to find out whether two regular expressions are equal, you need to check all strings below the maximum of the two respectice string lengths.

    But this idea only works for "real" regular expressions (in the computer science meaning), not for Perl Regular Expressions in general. So it will help to know if your regular expressions are regular or not.

    If they are regular in the CS sense, you can parse the regular expression and then convert it into a finite state machine. Then you "just" need to create all strings of length smaller or equal to twice the number of states, and check for those strings if they either match both or fail both regular expressions.

      What I want to do is really to parse the regex in some way and simplifiy it, if possible. So say I have a regex generator and for a certain input it produces

      .*(.*)$

      I would like to be able to recognise that this is the same as

      .*

      Thanks,

      loris


      "It took Loris ten minutes to eat a satsuma . . . twenty minutes to get from one end of his branch to the other . . . and an hour to scratch his bottom. But Slow Loris didn't care. He had a secret . . ."

        They aren't strictly equivalent, because the first regex has capturing parentheses even if they only will ever match the empty string at the end of the string. But assuming you have

        /(?:.*).*$/ # and /.*$/

        you could convert them into canonical form, that is, normalize all .* to become (?:.*), and then simplify, mapping (?:.*)(?:.*) to (?:.*).

Re: Testing regex equivalence
by aquarium (Curate) on Feb 21, 2006 at 13:07 UTC
    you could use "b" or whatever it is to compile a perl regex into token form. equivalent regular expressions (e.g. /.../ and /.{3}/ should end up the same. as for more complicated regular expressions, they may end up being equal depending on the dataset being fed in. the only way to test this kind of equivalence is to run both on the dataset in question.
    the hardest line to type correctly is: stty erase ^H
      equivalent regular expressions (e.g. /.../ and /.{3}/ should end up the same

      Evidently, you did not try this out yourself before posting. It took me 10 seconds to see that

      % perl -Mre=debug -e '/.../' Freeing REx: `","' Compiling REx `...' size 4 Got 36 bytes for offset annotations. first at 1 1: REG_ANY(2) 2: REG_ANY(3) 3: REG_ANY(4) 4: END(0) minlen 3 Offsets: [4] 1[1] 2[1] 3[1] 4[0] Freeing REx: `"..."'

      and

      % perl -Mre=debug -e '/.{3}/' Freeing REx: `","' Compiling REx `.{3}' size 4 Got 36 bytes for offset annotations. first at 3 1: CURLY {3,3}(4) 3: REG_ANY(0) 4: END(0) minlen 3 Offsets: [4] 2[3] 0[0] 1[1] 5[0] Freeing REx: `".{3}"'

      are quite different beasts. So that's not going to fly. Unfortunately, I don't have any better ideas myself. The only thing I can think of would be to take a look at japhy's Regexp::Parser, which will ease the pain of picking a pattern apart, but to put them back together again with an eye to checking for equivalence will be non-trivial.

      • another intruder with the mooring in the heart of the Perl

Re: Testing regex equivalence
by blokhead (Monsignor) on Feb 21, 2006 at 17:25 UTC
    See also this thread.

    The short answer is that you can do this by converting regular expressions to NFAs/DFAs (for more details, see my reply in the thread I referenced). However, this cannot be done in a super-efficient way. In fact, the problem of deciding equivalence of two regular expressions is PSPACE-complete (which is bad ;)). Even in your special case of deciding whether a regular expression accepts all strings, the problem remains PSPACE-complete (google for regular expression universality).

    Being PSPACE-complete does not mean it's impossible. It's quite possible, but will quickly become very impractical for large regular expressions..

    As for how to do the conversion to NFA/DFA, I can tell you that another monk and I are working on a formal language & automata toolkit that will make this easy, but it won't be ready for a while...

    Update: as for your idea above to "simplify" a regex, that is essentially the same thing and will not gain you anything. Testing regex equivalency reduces to the problem of finding the "simplest" equivalent regex (no matter how you define "simplest"), so regex simplification is still a PSPACE-hard problem (-hard instead of -complete because we're now talking about a search procedure, not a decision procedure).

    blokhead

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://531661]
Approved by marto
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others surveying the Monastery: (10)
As of 2014-04-23 20:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (554 votes), past polls