Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number

Re^2: Apply A Set Of Regexes To A String

by Cody Pendant (Prior)
on Oct 12, 2004 at 00:22 UTC ( #398341=note: print w/replies, xml ) Need Help??

in reply to Re: Apply A Set Of Regexes To A String
in thread Apply A Set Of Regexes To A String

Thanks Tachyon, I think I'm going to go with something like that solution.

If you've got a moment, can you say just a little more about why it would be particularly efficient, with regard to the "C code in the regex and hashing engines"? Thanks.

=~y~b-v~a-z~s; print
  • Comment on Re^2: Apply A Set Of Regexes To A String

Replies are listed 'Best First'.
Re^3: Apply A Set Of Regexes To A String
by tachyon (Chancellor) on Oct 12, 2004 at 06:11 UTC

    One of the main benefits of alternation is that you can compile the RE. Essentially an alternation RE is very similar to the loop, but the loop code has been optimised to the task and is in C with alternation, but generalised and less efficient if you do it in perl. The difference is significant, at least that is what this Benchmark shows.....

    use Benchmark 'cmpthese'; $iterations = 1000000; %re = ( foo => 'foo1', bar => 'bar1', ); $re = join '|', keys %re; $re = qr/($re)/; $name1 = "RE"; $code1 = << 'END_CODE1'; $_ = 'foo bar'; s/$re/$re{$1}/g; END_CODE1 $name2 = "Loop"; $code2 = << 'END_CODE2'; $_ = 'foo bar'; for my $sub( keys %re ) { s/$sub/$re{$sub}/g; } END_CODE2 cmpthese( $iterations, {$name1 => $code1, $name2 => $code2} ); __END__ Benchmark: timing 1000000 iterations of Loop, RE... Loop: 24 wallclock secs (24.77 usr + 0.00 sys = 24.77 CPU) @ 40 +377.94/s (n=1000000) RE: 8 wallclock secs ( 7.27 usr + 0.00 sys = 7.27 CPU) @ 13 +7551.58/s (n=1000000) Rate Loop RE Loop 40378/s -- -71% RE 137552/s 241% --

    But if I change that to a more real world situation by making the string a 14Kb one (approximately a web page size)

    $_ = 'foo bar' x 2000; Benchmark: timing 10000 iterations of Loop, RE... Loop: 20 wallclock secs (19.90 usr + 0.00 sys = 19.90 CPU) @ 50 +2.56/s (n=10000) RE: 30 wallclock secs (29.44 usr + 0.00 sys = 29.44 CPU) @ 33 +9.64/s (n=10000) Rate RE Loop RE 340/s -- -32% Loop 503/s 48% -

    And now the loop is faster. In fact try this case:

    $_ = 'fo ba' x 1000 . 'foo bar'; Benchmark: timing 10000 iterations of Loop, RE... Loop: 1 wallclock secs ( 0.71 usr + 0.00 sys = 0.71 CPU) @ 14 +064.70/s (n=10000) RE: 12 wallclock secs (11.10 usr + 0.00 sys = 11.10 CPU) @ 90 +1.23/s (n=10000) Rate RE Loop RE 901/s -- -94% Loop 14065/s 1461% --

    ~This is a purpose designed worst case for alternation as it requires continouous back tracking. So I have shattered my own delusions! Perl loops are faster than RE alternation.



Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://398341]
[Corion]: choroba: Yeah, I think that would be the good solution
[LanX]: I suspect the first string which comes from the DB ...
[LanX]: ... but this part is already in production for a year now
[Corion]: LanX: The "good" approach here would be to use the appropriate DBI parameters to make the driver decode strings properly. But that will have a ripple-on effect of messing up all the places where manual decoding happens ;)
[LanX]: which means albeit being broken UTF8 it'll be handled correctly
[LanX]: and the problem only occurs since we changed the emails to base64
[LanX]: my main problem will be to cnvince my colleagues that our productive code is broken oO ... so in the end I will just make a workaround :-/
LanX hates UTF8 for causing knots in his brain and stomach
[Corion]: LanX: Yes, that's the main problem - you have lots (and lots) of workarounds in various places and stages of the processing, and to clean that mess up requires action across the complete codebase. And it's almost impossible to do it piece-by-piece

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (10)
As of 2017-01-16 14:02 GMT
Find Nodes?
    Voting Booth?
    Do you watch meteor showers?

    Results (150 votes). Check out past polls.