In theory, Module::Build should respect whatever the current Perl tells it about module locations. Do you maybe have a leftover .modulebuildrc somewhere that screws up your configuration? Are there maybe Module::Build-specific environment variables?
Corion, you're an absolute star - I forgot about .modulebuildrc but it turns out that I do have a very old one and that it indeed points install_base to my home dir. I would never have solved this in a million years, thank you so much!