Re: Testing: Fun for the family

[sfink]

I have been doing a lot of test-driven development recently, and so far, I have to say it's a mixed bag.

It is enormously valuable to be able to have an actual demonstration of the correctness of a subsystem, rather than relying on gut feel and developers' memories of what they have "tested". It is pretty much impossible to have any confidence in a release unless you have the tests to show that at least a subset of the functionality is fully working -- without the tests, it's amazing what can be broken and still appear to work for the things that have been thrown at it so far.

However, and perhaps this is more a reflection of me than the methodology, but I have not had great experiences with using test-driven development to drive the design. Whenever I start out by writing the tests and then making the design work for as many of those as I can, I end up with a simplistic design that:

• works very well for the core tests that I was most concerned about
• adequately for the remaining tests that weren't such a high priority
• and inadequate for the various new bits of functionality that inevitably get added to the requirements as the project progresses (i.e., the new tests that weren't written when the design started).

I always seem to end up in the same cycle: I get everything working and passing all or almost all of my current tests, then either get a new requirement or just expand the project to include more of the pieces of the overall spec. I then discover that some fundamental piece of my design needs to be changed slightly. Theoretically, this should be no big deal since I have the test cases to allow me to change things drastically without worrying about breaking something subtle. In practice, that is true -- but I end up spending far too much time making the change, fixing things up so that all the old tests work, then making use of the change to implement the new tests. Lather, rinse, repeat.

I find that if I step away from the details and create the design, and even part of the code, based on a wider view of what the system needs to do know and might need to do in the future, that I end up with a much more robust system that can pass the tests and can deal with the inevitable late-stage additions.

This opinion is probably not too popular with the current conventional wisdom about agile development processes or whatever, but I'm just reporting what my recent experience has been. I am more productive with initial design then initial coding then tests then final coding. And again, this may just be due to defects in the gooey stuff between my personal pair of ears, but I'm wondering if others have experienced the same.

[rkg]

I am a big fan of TDD -- and I agree with sfink's post. In particular,

I then discover that some fundamental piece of my design needs to be changed slightly. Theoretically, this should be no big deal since I have the test cases to allow me to change things drastically without worrying about breaking something subtle. In practice, that is true -- but I end up spending far too much time making the change, fixing things up so that all the old tests work, then making use of the change to implement the new tests.

Right on.

The refactoring and XP folks make a big distinction between refactoring (improving the design of existing code) and adding new functionality. And often it is claimed you can refactor safely because 'your tests protect you'. Agreed, but like sfink, I find refactoring requires you to recode (often nearly all) the tests as well.

This recoding is well worth it, of course. And TDD as a whole is very much worth it. But still, the tests -- necessarily so -- are very dependent on the specifics of the code tested. I find my tests break and need rebuilding (not just fail, and require the module to be changed) when the module goes through large revision.

My current testing challenge is adding application or user tests -- so far I've been living the easier world of unit tests.... app tests (particularly in a web environment) take more effort. I'm plan is to continue using extensive unit testing to confirm each module behaves as it should, then confine my app tests to one or two simple basic run-thrus of the web app's main funtion. I've noticed when the apps break, it is no longer due to the objects (which are well tested), but to the thin (Mason) glue holding them together. That said, I've not done much app test writing yet, and (unklike unit tests), can't seem to get into the rythm of writing app tests.