I often have to update Perl and the CPAN dists i use for my projects on many machines at once. Over time i have to come more and more to realize how much of a drag badly written test suits are. In this post, i'll try to formulate this into a somewhat readable list of things that can be improved.
Make your tests as short and fast as possible
Some dists really seems to drag their feet when it comes to running tests. Do you really need to run a gazillion slow tests on every single computer your dist is installed? This wastes the users time.
For example, do you really need to test if 1+1=2? You can assume that Perl did some extensive tests on the basics during its installation, so you don't have to.
Don't run extensive "timeout" tests unless you absolutely have to.
Make more of your tests Author-only tests
Decide which tests only makes sense for you, the author. Prime examples would be tests that check the documentation or running Perl::Critic. Others might be check if your network protocol handler conforms to the specification. Unless part of your code is specific to the system architecture, these tests only need to be run whenever you change the code.
Don't make assumptions about the users network architecture.
Quite a few dists in CPAN make assumption on how the network setup of a users should behave. Don't assume that the user is even allowed or able to access the internet. For the same reason, don't assume DNS resolution is going to get you the expected results, these days many users and companies run filters on Nameservers.
Don't try to access servers you don't personally own or have explicit permission to use
I've seen tests that run against other peoples (or companies) servers. This is generally frowned up on, you are wasting resources of the server owner and quite possibly their money.
Also, your tests might suddenly start to fail. You don't have control over those servers. If their configuration suddenly changes or they go offline, your users might have problems installing your dist.
Accessing certain sites may also be against company policy or even against the law in the country the user resides. Think "news sites" for example. Accessing news sites during work hours might get the user into trouble. Trying to access these sites might even get the user into legal trouble because they are "banned" in the users country.
Don't ask stupid questions while installing
Quite a few dists hold up the installation process to ask stupid questions. This is very annoying, especially if you run the installation on multiple hosts and once and you have to constantly switch through all tabs of your terminal. Just to check if some installation script has put its lazy feet on the table and won't do anything until you press enter.
Like "should i run the network tests over the internet". Answer: "No, you shouldn't" (see above). Provide some Author-tests instead that the user can run if they run into trouble - and document this in the dists "Troubleshooting" section.
Another good example is Template Toolkit. You get the questions "Do you want to build the XS Stash module?" and "Do you want to use the XS Stash by default?". How the f should i know? You are the author and you should know the answer to that better than me. If the author answer is "i don't know", then try to build the module while catching errors as non-fatal. If the module build, run its tests, if those tests work, use it as default.
Don't keep outdated tests and workarounds for external modules
If your dist uses external modules, and you encounter bugs, you will probably check for those bugs and write a workaround. Say the author of that module has since fixed the bug. Instead of keeping slow workarounds and extensive tests for that outdated module around, just require the current version of that external module as your minimum version.
Don't install if your requirements are not fulfilled.
If your dist needs either ACME::Foo or ACME::Foo::PurePerl installed to function properly, don't just print a message and continue. Either fail the installation (not good) or pick one at development time. In this case, picking the PurePerl version might be more reliable. Just printing a message during installation is pretty useless, especially if your dist is getting installed at the same time as a number of other dists. Your message might be on screen for only a fraction of a second.
Don't make the computer unusable during building and testing (unless you have to).
A good example where this is unavoidable is install Tk. It has to pop up many different windows to test this GUI library, which makes working on the computer while the installation is running impossible due to focus stealing. But if at all possible, avoid this.
Be careful when testing floating point numbers
Different systems might handle floating point numbers in slightly different way. Just because 1.0 + 0.05 = 1.05, this might not be true on the users system. It might only be "1.04999999". That is how floating point numbers work on on computers, because they use binary representations of varying length (precision) to work on those internally.