Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl-Sensitive Sunglasses

Re^2: Analysis of Regular Expressions

by PetaMem (Priest)
on Mar 17, 2010 at 17:16 UTC ( #829227=note: print w/replies, xml ) Need Help??

in reply to Re: Analysis of Regular Expressions
in thread Analysis of Regular Expressions


ok, it's clear that solution 1) has exponential runtime and space requirements (size of the charset doesn't even matter - (well until JavaFan points out that a charset of 1 char would...) )

I basically went for solution 2) before, but only with some half-baked heuristic constructs, so probably a more high-level description of the problem is in place: I'm working on a chatbot demo which has to catch some phrases.

Thats a simple pattern matching just before any parsing, and delivers decent speed. And the whole point of this discussion and my endeavour is, that the set of the regular expressions gets extended and it is not feasible (at least I haven't come up with a feasible way) to sort them by "do-what-i-mean" manually.

Doing a trivial "sort by length" has actually worked very well for a set of up to 500-600 rules. But now I start getting false matches ONLY because some "more generic" regular expressions are handled before the more specific ones.

Under the assumption, that (a|b|c) is more generic than (a), it is clear, that the simple "by length" ordering will result in the wrong order, because I want to process the "specific" rules first, and the "generic" last.

What has helped:

  • removing e.g. named matches from length considerations, making e.g. (?<name>...) 9 chars shorter
  • Having an alternation like (a|b|c) being considered as just having length of 1, same for character classes
  • actually subtracting a length of 1 for every wildcard.

Maybe the best solution could be to give every rule an identifier (which has happened anyway), and add some information "try this before ruleX". Which is starting to look like a parser. :-)

    All Perl:   MT, NLP, NLU

Replies are listed 'Best First'.
Re^3: Analysis of Regular Expressions
by Ratazong (Monsignor) on Mar 18, 2010 at 07:02 UTC

    Hi PetaMem!

    Thanks for that additional information! Chatbots are a really interesting topic!

    Looking at your specific problem (and not the general generalicity of regexes) makes me wonder if the empirical approach (do a check on how many possible strings are matched by the regex) really doesn't work. However with some modifications:

    • Your problem doesn't seem to be real-time - and the order of your rules seem to be static:
      therefore you could do the generalicity-rating in some offline preprocessing phase, e.g. by letting your computer work on it all night/weekend/holliday
    • Instead of looking at all possible input strings, just narrow it to the expected data. You will probably have tons of chat-logs which you can use
      So your rating could be: how many matches are found by a regex in a given set of logs

    Regarding the high number of possible characters (as also mentioned by JavaFan): Here you could do some preprocessing, e.g. by replacing the german A-Umlaut by ae. This will also help with some other "languages" like leetspeak ... your chatbot seems to get confused when greeted by a friendly "h3110" ;-)

    However I fear that any automatic ordering would just be one criteria for determining the order of the rules. You will probably add additionally some rating done by a human. And the idea of giving additional context to the rule (like you wrote: try this before ruleX ...) sounds great to me. Have you also experimented with randomness? (Using a random order in case of several rules have a similar rating.) That way the answers might not be so predictable...

    HTH, Rata

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://829227]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (7)
As of 2018-06-21 22:03 GMT
Find Nodes?
    Voting Booth?
    Should cpanminus be part of the standard Perl release?

    Results (119 votes). Check out past polls.