Re: Test Case Generator

[sundialsvc4]

As you can imagine, building a really good test-case requires thought, not automation.   You usually need to work carefully with the developer, but in the role of “the devil’s advocate who came from Missouri.”

Perl has definitive test-suites such as Test::Most, which of course means that the tests are themselves programs ... so you can use loops and such in writing them.   Every CPAN module includes its own test-case suite (generally, the file-names end with “.t”), which you can definitely should use for ideas and examples.

IMHO, there are three or so things which a good test-suite should definitely cover:

• Normal Cases:   Every single one of them, every variation that could be encountered under normal conditions with valid inputs in normal operational situations.   Build up your tests from simple unit-tests of expected atomic behaviors, to more complex scenarios involving units working correctly together (which by that point can be confident that the primitive operations they are invoking do work properly).
  

• Edge Cases and Error Cases:   The computer, itself, must be the guardian of its own correctness, because there is no one else who could possibly do so.   Test the boundaries between right and wrong:   look for that “<=” condition that was only supposed to be “<” ... divide by zero and see what happens.   Stuff incorrect and al-most incorrect inputs in, and see what happens.   Write tests that explicitly determine what should happen in those error cases:   the test fails if it doesn”t throw an exception, or if it’s the wrong one.

• Sh*t Smoke Tests: These are tests which try to simulate what happens when stuff is hitting the fan; that is to say, in actual service.   These tests cover every user-accessible or client-accessible control or API, every use-case, and (in the case of UI testing) they do so in an unpredictable but legitimate sequence.   In particular, they are looking for holes.   The things that have a 100% probability of occurring when Steve Jobs (RIP), Steve Ballmer, Larry Ellison or Bill Gates is performing a demo for the world.

The developer can’t be the tester.   It takes a different mind-set, I think.   People definitely trend either toward “writing it” or “blowing holes in it.”   In any case, whether consciously or unconsciously, a developer’s own self-written tests (although she should be asked to provide some ...) won’t be complete and will tend to be too kind.   The relationship between the two of you of course is not adversarial; you are working toward a single mutual goal, and if both of you do it well, that pager will remain silent all night long.   (“Priceless™ ...”)

I think that the job of the tester/QA team is really the most important job in the world, because, as I said, only the computer itself can be the guardian of its own correctness.   The hardest thing about any flaw is finding it.   New flaws can will be introduced by anyone at any time.   The only assurance that you have that the code is still correct is that battery of tests ... which, as you can see from significant CPAN modules, might number in the thousands.   (If you want to know the major reason why the Perl system is thought of as a reliable bastion tool of this industry, you’re looking at why.)

[eyepopslikeamosquito]

Smoke Tests: These are tests which try to simulate what happens when stuff is hitting the fan; that is to say, in actual service. These tests cover every user-accessible or client-accessible control or API, every use-case, and (in the case of UI testing) they do so in an unpredictable but legitimate sequence.

(Some pretentious typography elided). Do you have a reference that supports your definition of smoke test?

From Smoke testing (wikipedia):

In computer programming and software testing, smoke testing is a preliminary to further testing, intended to reveal simple failures severe enough to reject a prospective software release.

From stack overflow question:

Smoke test: A simple integration test where we just check that when the system under test is invoked it returns normally and does not blow up. It is an analogy with electronics, where the first test occurs when powering up a circuit: if it smokes, it's bad.

From SearchWinDevelopment:

Smoke testing is non-exhaustive software testing, ascertaining that the most crucial functions of a program work, but not bothering with finer details.

Update: some more definitions from the stack overflow question:

Smoke testing: first tests on which testers can conclude if they will continue testing.

Smoke testing is done as a quick test to make sure everything looks okay before you get involved in the more vigorous testing.

[RichardK]

Of course the developer can be the tester -- What about test driven development?

If a developer isn't able to test their own code then they're not think logically about what they are doing.