You are using the $& special variable in your program. perlre warns:
WARNING: Once Perl sees that you need one of $&, "$`", or "$'" anywhere in the program, it has to provide
them for every pattern match. This may substantially slow your program. Perl uses the same mechanism to
produce $1, $2, etc, so you also pay a price for each pattern that contains capturing parentheses. (To
avoid this cost while retaining the grouping behaviour, use the extended regular expression "(?: ... )"
instead.) But if you never use $&, "$`" or "$'", then patterns without capturing parentheses will not be
penalized. So avoid $&, "$'", and "$`" if you can, but if you can't (and some algorithms really
appreciate them), once you've used them once, use them at will, because you've already paid the price. As
of 5.005, $& is not so costly as the other two.
So avoid using $& and friends. A simple workaround is to put parens around the whole regex, and use $1 instead.
Depending on your perl version, perlre might offer other workarounds.