You don't mention which version you ran your test on, but if it was 5.8, part of the reason could be that glob now returns the files pre-sorted (according to perldelta).
However, I think that tye was alluding to that the OP's code shows not just reading list of files, but looping over them, opening and reading each one in turn. Assuming that the opening and reading part stayed the same, 2 or 3 seconds on generating the list is unlikely to make a significant difference overall.
Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller