if they use any third party software (comparable to CPAN modules) they know the status of every package, if it supported on their platform or not
That's just not true. Only the most popular, best supported open source Java projects actually tell you in any clear way which versions of Java they are known to run on. They don't have anything as complete as CPAN Testers, i.e. they assume that Java 1.4 will work identically on every JVM on every platform (and good luck if you believe that). CPAN Testers provides actual test reports for specific combinations of operating system and perl version.
The most popular CPAN modules, like DBI, specify which versions of perl they support. In addition, since CPAN modules usually come with tests that you can run on your own platofrm.
There is certainly more that could be done, but claiming that the automated testing system for CPAN is somehow lagging behind Java projects' totally ad-hoc and often non-existent version requirements is just wrong.