I would concur with the comment regarding generalizing to arbitrary dimensionality. I don't the the generalization would be all that difficult and the payoff would be worth the effort. In terms of the changes that would be required, other than the implications for your algorithms

I think that the arguments to new() would most appropriately be an array ref

A size() method would be useful both internally and externally

You would have to have some internal exception handling for your overloaded operators in case, for example, someone tried to subtract vectors of different dimension

My only other comment is that I think the namespace should stay Math::Vector rather than your alternative, Physics::Vector. After all, the concept of a vector is defined precisely in mathematics and only used by physicists. (Athough the mathematical definition does not necessarily concern vectors of numbers, I don't think you'll confuse anyone.) So the designation in the Math::* namespace is entirely appropriate.

Excellent contribution! I look forward to using it.