Is it in the phalanx?
I think this probably the most important point. A quality label for modules would be VERY helpful as ratings obviously don't work. I don't know much about the phalanx project. But a project where authors and users can submit modules easily and get objective advice from experienced developers (maybe bored during coffee break) would be great. The review process should be very simple, maybe with a standardized scoring table. Once the module passes the quality requirements, it should be accepted in phalanx or something similar and labeled in CPAN accordingly.