Re: Recommendations for efficient data reduction/substitution application

[LanX]

I solved a roughly similar problem using the trie optimization of the regex engine.

You need to put the different match clause into a long or-chain.

The replace-part has to identify which regex matched.

Further more you should benchmark how expensive replacements are on longer chunks.

A sliding window technique could be the answer.

Hmm... Could you identify where the bottleneck really is by benchmarking match and replace independently?

[BrowserUk]

Hm Did they fix that "As it seems that trie optimization brakes for more than 15000 alternative patterns." yet?

C:\docs\OriginOfSpecies(Darwin)\2009-h>\perl5.18\bin\perl.exe \test\10
+43602.pl 2009-h.htm
 Finding 203474 words (of 216808) took 0.173504 seconds using a hash
 Finding 203474 words took 2072.099258 seconds using a trie(via regex 
+engine)