Wanting to only bump module version on API change sounds like a different (albeit legit) approach from "semver" (semantic versioning). I describe that as incrementing when (from least significant to most significant digit) you fix something, add something, break something. That approach would dictate that a version bump of some sort is recognised somewhat as a good idea even if one only makes an improvement to a module, without API change.
As a sidenote, I haven't felt the urge to use semver for PDL. If the major version incremented on every breaking change, it would probably be roughly on major-version 30 by now (including from before my time as maintainer).