Marc Lehman states the problem pretty well, if you want to call it perl instead of debperl, you should not break it by removing .packlist files , so this is a debian bug, and they should fix it
If it was just about not removing the files, that'd be easy, I solved that part in 15 minutes on Gentoo. Alas, the files are not even installed by a standard Makefile as produced by ExtUtils:MM when using the vendor_install target, so it doesn't seem only Debian's problem but intended behavior---with ExtUtils::MM being a core module I presume even some sort of official Perl policy. I'd be interested in why this is so and if I can/should circumvent it.
Edit: so far every single vendor I've looked at (OK, just Linuxen: Gentoo, Debian, Ubuntu and CentOS) doesn't package the packlists. If someone could point me to one who does so I can look at their policy and build system, that would be enormously helpful already.