in reply to When Test Suites Attack

I've been finding more and more that small changes in my code are making huge changes in the output and trying to continuously update the tests to exactly match...
I don't know your problem domain, but would it be possible to test for other (higher level) invariants besides an exact textual match? For instance, maybe you could just check that a transformation preserves syntatic correctness, or that all tags are balanced, or properly nested, or conserved, etc. Let's take a different example. If you are creating a ray tracer, you could have a unit test that renders a red sphere on a black background and then compares that bitmap to a correctly rendered red sphere bitmap, checking that each pixel is correct. Or, you could instead write tests like... Does that make any sense?

Replies are listed 'Best First'.
Re^2: When Test Suites Attack
by Ovid (Cardinal) on Oct 28, 2005 at 22:32 UTC

    I'm taking a roughly similar strategy now. Previously I had many of these:

    is_xml $result, $expected, '... and we should receive the correct XML';

    By generating $expected with a different algorithm (to ensure that I'm not just reaching into buggy code), I had to constantly maintain two different sets of code which did the same thing. It was very painful. Now I'm doing this:

    is_well_formed_xml $result, '... and we should receive well-formed XML';

    And to verify that it's correct, I'm adding more high-level integration testing. It's an annoying trade-off, but like what you're suggesting, it's a reasonable one.


    New address of my CGI Course.