Re: [RFC] Building Regex Alternations Dynamically

[LanX]

it's almost TL;DR, so two separate posts might have been better. :)

Just three remarks:

• IIRC there is a hard limit for the length of a regex-string ... (or was it only the repetition counter .{x} limited to x aprox 32000 ?)
• "newer" Perls come with something called trie-optimization which will considerably speed up such or-clauses.
• Trie-optimization has definitely a hard limit, where it falls back to (much slower) old behaviour.

[haukex]

Hi LanX,

Thanks for your comments, I'll think about whether I can provide a TL;DR version - my thought was that impatient users could just copy-n-paste my code :-)

IIRC there is a hard limit for the length of a regex-string

I'm not sure, I'll have to look into it, but I do know this works :-)

my $word_regex = do {
    open my $wfh, '<:utf8', '/usr/share/dict/words' or die $!;
    chomp( my @words = <$wfh> );
    close $wfh;
    my $re = join '|',
        map {quotemeta}
        sort { length $b <=> length $a or $a cmp $b }
        @words;
    qr/$re/i };
print length($word_regex), "\n";

__END__

964836

Thanks,
-- Hauke D

[LanX]

> > IIRC there is a hard limit for the length of a regex-string

Maybe not in the string length but for sure in the processing.

something like /x{33000}/ will result in

Quantifier in {,} bigger than 32766 in regex; ...

But that's not exactly relevant for your case :)

[haukex]

Hi Rolf,

IIRC there is a hard limit for the length of a regex-string

Other than RAM, I haven't found one yet, I just built a qr// regex from a 1GB text file without alternations and a second from 40MB of text with over 10M alternations.

As you may have seen I added a TL;DR, thanks for the idea!

Regards,
-- Hauke D

[LanX]

Yeah sorry I had

> > (or was it only the repetition counter .{x} limited to x aprox 32000 ?)

in mind. Since regexes are compiled to op codes hard limits are unlikely.

> As you may have seen I added a TL;DR, thanks for the idea!

Hmm probably it's time to include support for TOC = table of content in my wiki add on :)