First of all, you're talking about a time-difference, but there's no context. The difference between 0.01 and 1.01 seconds is far more interesting than the difference between 15h36m55s and 15h36m56s.
Second, your first talk about a difference of 3/4 of a second, then of a difference of 5. There's quite a difference in differences.
Third, without the return, you're doing 10 comparisons in an otherwise almost empty sub (the first time the sub is called, memory needs to be allocated, but Perl doesn't throw it away, making that in any subsequent call to the sub, the assignment to $a is relatively cheap), so I do expect there to be a difference.
Fourth, use the NYTProf profiler, and see how much time each statement takes.