Your opinion is of no consequence. What matters is the opinion of the user. If you don't try to tell it where to install, then under *normal* circumstances (ie, if the user is using a perl that is in his path) then it will install in the path, probably in /usr/local/bin or /usr/bin. Which is what the user wants. Under other circumstances, such as when the user is using some other build of perl, then your code will not unexpectedly pollute the directories in his path.
I don't know whether Module::Build will let you force your code into particular directories. But just because it lets you doesn't mean that you should.