Perl doesn't use a power-of-two size extension mechanism. When perl increases the size of an array's cache, the formula is generally:
new_top_index + (allocated_elements / 5)
For a push, top index is just the allocated size plus one if we've triggered an extension, so when pushes blow past the end of an array they make it 20% bigger. There's a lot more memory copying going on in that case than with a power-of-two extension system. (For folks following along at home, this formula isn't always used, but it is for the case in question)
Given that the system in question is a Windows box, it wouldn't at all surprise me to find that this is the problem, with the big speed hits coming as the array is resized and ever-larger chunks of memory have to be allocated from the system and copied into. The individual pushes are probably running at full speed, but with ever-increasing performance hits for reallocation you'd not necessarily notice that.