It also avoids wasted effort. What's the point of expending a non-trivial amount of effort to ensure that all my CPAN releases work on 5.002, when there's a fair chance that no-one will ever need it? I do not have the time.
On the other hand, if I received a cry for help from a lonely programmer who was battling to install one of my modules on 5.004, and gave me a report of what the offending lines were in the code, there's a better than middling chance that I just might go ahead and do it, because I would be happy to help this person out.
Although I could see the desire to have your code be used the world-over, sometimes the cost of making things backwards compatable makes them more obfuscated and harder to maintain, or result in other tradeoffs; yes, it is possible to write things in Perl 4, but many of us enjoy the added features of Perl 5.8 and would love to see the new features of Perl 6.
I'm going to have to go against the grain on this one, and say that rather than trying to make all of the modules used behave like older versions of Perl, it'd be better (although I'm guessing vastly more complicated) to make a source filter that allows a given file to be parsed as the older version. Of course, the problem is that this would likely cause slower execution time and additional overhead.
I don't think it's a good idea to try to tie our hands and insist backwards compatability by changing the modules is the best way to go. We have newer tools, and we should use them. Decisions were made to maintain older versions of perl, and this is one of the tradeoffs. Yes, it sucks, but it's their decision on upgrading perl, and this is just one of many factors that might influence it.
* Note -- I don't have modules published through CPAN currently, but I have had to deal with management who wouldn't allow me to upgrade code that needed it -- I just went and installed multiple versions of perl, and made sure that the libraries I wrote worked in both versions. The only problems I remember having was some quirk with Net::LDAP, and I just went and got an older version, so the two matched, as management didn't give me enough time to analyze what had gone wrong