I suspect that people's opinions on this debate are highly correlated with whether or not they have worked in an environment where they were forced to use an older version of Perl (or other similar package). The attempts to get newer things working on such machines leave scars that go very, very deep...
There are many very good reasons why different organizations may need to use older versions of Perl, and yet still need newer modules. This fact in no way makes module authors responsible for supporting older versions, although I would hope that it's enough to make us accept nonintrusive patches to provide such support.
The way I look at it is that if you're putting a module out there, you are doing it to help people. Maybe that's not your primary concern; that might be the ego gratification or whatever, but you don't get any of the other benefits without actually helping people. So the question of maintaining backwards compatibility is strongly influenced by your estimate of the number of people such support would help. For example, a module in a ton of other modules' dependency trees has much more motivation to provide backwards compatibility than a leaf module.
My personal approach is to try to maintain backwards compatibility, up to the point where using a new feature would make the coding appreciably easier or better. When that happens, I discard support for older versions lacking that feature without a backward glance. Thus, I could say that I only break backwards compatibility when there's a decent reason for it, but in truth, the reason doesn't have to be that good. If more people used one of my modules, though, I would worry a lot more.
I'd also like to point out Python as a counterexample. I tend to work with somewhat but not excessively old Linux distributions, and I've lost track of the number of times I've run into "application A needs python 2.2 but also library L which I have for 2.2 but I also need for application B, which requires a newer version of library L that requires and installs into python 2.3's tree". For whatever reason, I am constantly fighting version dependency battles with Python, and rarely am with Perl. Some of that is because Perl makes it relatively easy to support older versions -- even when a major release is technically not API compatible, Perl tends to minimize the API changes to the extent that very little code actually breaks.