in reply to
What goes in a test suite?
but what would go in a test suite?
Anything that is important for your product. Here are a few points
you may want to think about. Not all points are relevant in all
Correctness is usually important, and one of the first things that
are tested. This could range for simple things like "if I plug X and Y
in this function, the result should be Z" (typically the things that
are tested when "make test" of a CPAN module is run) to "if I push
the red buttons after I disabled the blue switch, while standing on
my head humming the national anthem, does it make a decent decaf".
System resources. How CPU intensitive is it? How much memory does
it use? If your product has always hummed along on a Sparc 5, but
a new minor release suddenly requires a 6 processor 3800 than your
customers won't be happy, even if the product passes all correctness
tests. Besides CPU and memory usage, there's disk usage, I/O usage,
shared memory usage, etc, etc to consider.
Robustness. What does your program do when something unexpected
happens, like a file that suddenly disappears, a network that goes
down, a remote server that no longer answers, a disk that's no longer
repsonsive? How does the program behave when it receives signals?
What does it do when I cut-and-paste a million characters in a small
text field? How does it behave on garbage input data? Or on no input
data? What happens if the program crashes?
User interface. If you have a GUI program with many screens, is the
user interface consistent, or does each screen need its own manual?
Is the user interface simple? Can I do things that need to be done
often with one keystroke or mouse-click, or do I need
control-alt-shift-meta-cokebottle all the time? Does your product
confirm to the house style? Is each background the right shade of
orange, or do you still have screens in green, the house style of
2001. Are all the logo's correct? .
Regression tests. Adding a feature in one part of the software
shouldn't have an effect on another, unrelated, part of the software.
You can lose customers that way, because people do get upset if the
software suddenly behaves differently. I'd be mighty upset if the
next release of vi suddenly gave a different meaning to 'a'.
Stress testing. Testing a single instance of a product is one thing,
but what happens if two people use the product at the same time?
Or 10? Or 1000? How does the product behave if run continuously for
24 hours? Or a week?
Security. Can the software be used to comprise security in one way
or the other? This not only means gaining unlawful access to the
box the machine runs on, but also access to data (for instance, in
 Don't laugh. I've worked for a company that was once
threatened with legal action because a button displayed the old name
of the product - a name that our company no longer had the right to