|The stupid question is the question not asked|
McMahon's scratchpadby McMahon (Chaplain)
|on Jun 01, 2004 at 16:39 UTC||Need Help??|
I do what we call "functional testing" or "system testing" or "end to end testing" or "black box/grey box testing". When I talk about a "test harness", I mean a system that allows the user to design test cases that the harness will run and validate automatically.
The best ones I've seen (or built, in one case) drive a system not from the UI, but directly from a set of data like a spreadsheet.
First, though, I'm going to stipulate that a "test case" must contain 3 elements: an objective, a procedure, and an expected result. Only the last two can be automated.
So a trivial test plan would consist of test cases that have objectives:
1) Insert service order
2) Change service order.
3) Change service order multiple times.
4) Delete service order.
So we'll keep the data for the tests in a simple spreadsheet:
For each action, the harness will check the database and report PASS or FAIL depending on whether what's in there matches the values in "ER", the Expected Result. Note that this lets a business user design tests without knowing the underlying implementation of the tests. (BTW, a database query may not be the best way to describe Expected Results. The most complex harness I ever built did a massive diff on sets of files, rather than a surgical query for data.)
Some other considerations: for tests like these, it is imperative that the test environment be resettable to a known state before the tests run. Part of the harness is a controllable system.
So that's an overview of what I know about building test harnesses.