Re: A danger of test driven development.

[BrowserUk]

I think the problem with TDD is essentially the same problem as with any other development methodology, that of overemphasis of one aspect of the development process with respect to other aspects. When the tests become more important than the code being tested, you have a problem.

However, this problem is not a fundemental problem with the methodology, it is a problem with either the implementation or the day to day execution of it. Or both.

Just as an overzealous adherance to OO doctrine can lead to the disguising of an inherently global entity behind a design pattern that trades a the simple global for a complex one with the consequence of greater complexity and no real benefits. So, an overzealous adherence to TDD can lead to the situation where the test suit takes on a greater importance than the code under test.

One of the signs of this overzealous application of a methodology is when the important part--ie. the code--starts to be designed around or altered to accomodate the methodology. Another is when the final purpose of the code becomes subordinated by the goals of the methodology. When "100% tests passed" or "100% code coverage acheieved" goals take a higher priority than "the code functions within specification", the real goal has been lost and the problem has arisen.

It is also important to realise that fixing the problem means adjusting the methodology or it's implementation, not throwing it away in favour of some new "magic bullet".

If there is one lesson that history can teach us with respect to code development, it's that being taken in by the promises of the latest, greatest, buzzword-complient paradigm to the exclusion of previous hard-won experience and common sense, invites history to bite us in the arse as we re-learn the forgotton lessons.

[Roy Johnson]

When "100% tests passed" or "100% code coverage acheieved" goals take a higher priority than "the code functions within specification", the real goal has been lost and the problem has arisen.

The tests are supposed to be a translation-into-code of the specification. TDD is therefore supposed to be S(pec)DD. Obviously, that doesn't always happen.

What TDD does is shift the quality assurance constraint. Instead of looking at the program, you look at the tests. Bad things happen if you think that quality assurance is no longer an issue. It's every bit as important as it is in any other paradigm; it's just supposed to be easier to verify.

[jplindstrom]

One of the signs of this overzealous application of a methodology is when the important part--ie. the code--starts to be designed around or altered to accomodate the methodology.

Actually, in this case the fact that the code is designed to accomodate the methodology is a very good thing. Because code which is the result of tests/TDD tend to have low coupling.

Normally there is only one scenario in which code is used; the application. That means there was little outside pressure to steer the design.

When doing TDD, there is an additional scenario in which the code must work; the tests. This forces the code to be useful outside of the application, in the fairly scarce environment that is the test suite. The code must be decoupled from the rest of the application, because... it's not there.

The beauty of this is that you don't have to be a brilliant designer who is attentive and clever all the time in order to produce well structured code. You just follow these small steps one by one (write test, write code until tests pass, remove duplication, goto 10) and the rest ends up being an emergent property of those steps.

Much appreciated by us bears of small brain.

/J

[Aristotle]

The goal of TDD done correctly is to have a specification fully codified as a test suite. The benefit is that unlike prose, code can be debugged.

On the way there, the code does indeed end up accomodating the test suite, but that is a good thing: experience shows that code which is hard to test is quite simply hard to use.

Obviously writing tests just for the sake of writing tests is pointless, but that’s a truism.

[BrowserUk]

I was not attempting to critique TDD, I think the concept is a good one.

When I said "the problem with TDD" in my first sentence, I should have stuck with the OPs word, "the danger with TDD".

As I attempted to make clear in the second paragraph, "this problemdanger is not a fundamental problem with the methodology", or any other methodology, it is a problem of mis-application, or mis-emphasis, by some people, sometimes.

It is a matter of balance. If the balance swings to far in favour of one aspect of the development process, then others get watered down or totally omitted.

In the case of TDD, is easy for the programmer to write tests for the code they are going to write, instead of a test that encapsulates the specification. If their understanding of the specification is correct, then the code they intend to write will be correct and the tests will be correct.

But if their understanding of the spec is incorrect, then they write tests to test the code they intend to write, and from that point forth, code is verified against test and test against code, and the spec is forgotten until some third party attempts to use their code in conjunction with the spec.

Too many practitioners see the tests the programmer writes as the conclusion. They make great unit tests but very poor systems and integration tests.

The best test of an API, is to write an application that uses it. From the spec and preferably by someone other than writes the code that implements the API, or the person that specified it. But the latter is preferable to the former.

Having an application that uses the API is the surest way of ensuring that the API works for that application. Not just that individual functions and methods perform to requirements within the ranges of parameters that the application calls them on. But also, and perhaps more importantly, that the separate parts of the API work together and work with the rest of the application.

That is, not just "work" as in given parameters within specified ranges, produces the correct results, but work as in "is a good fit with the other functions within that particular API, and works with any and all other APIs and data representations that the application uses".

So you don't have one API that wants it data represented in relative format and another that takes absolute, forcing the application to constantly switch between formats when passing data between APIs. (Often as not, the one taking absolute format data will immediately convert it to relative format internally, or vice versa).

Eg. one uses DOB, and the age; or window coordinates versus screen coordinates; imperial units versus metric; and so on.

Once written, an application that uses an API will not only test for this type of impedance mismatch which is rarely tested by unit type testing (how many times have you used or read about APIs that "work fine but are a bitch to use"?). It will also tend to test the required range of parameters (and combinations thereof), of the individual APIs.

This will often tend to be a subset of the full range of possibilities that the API could be called upon to handle, but unless and until another application comes along that needs a wider set of (combination of) parameters. But if that second application never comes along, it can be an expensive exercise to write code and tests (or tests and code:) to handle them.

(Silly)eg. Writing a square root function to handle negative numbers and produce complex results would be an expensive option if it is never called upon to do so. Moreover, doing so could add further to the problems of the applications using it that would be better served by an exception being raised if passed a negative number than being returned a correct but useless imaginary result.

None of this means that TDD is a bad thing, only that over-emphasis of one aspect to the detriment of others is bad. "Oh! We'd never make that mistake", is everyones first reaction to the type of exaggerated examples I've cited, but they do happen.

Take the Mars Orbiter problem.

"This is an end-to-end process problem," he said. "A single error like this should not have caused the loss of Climate Orbiter. Something went wrong in our system processes in checks and balances that we have that should have caught this and fixed it."

Or the Short sighted Hubble telescope, the most perfectly flawed mirror ever produced up to that time.

Hubble is working perfectly but the Universe is all blurry.

TDD makes a virtue of having the programmer write the tests against which his code is tested. It works fine, provided that his myopic viewpoint is balanced by other techniques that give overview.

And that is the danger I am alluding to, not only with TDD, but any methodology that is given too high an importance to the exclusion of balance. If 100% test passed and 100% code coverage is not balanced by fitness for purpose, ease of use, good impedance matching with it's foreseen applications; if unit testing is not balanced against actual or "typical usage" systems and integration testing; then the overall result can be an expensive, but very well tested disaster.

It doesn't matter how many tests you write, or that they all pass, if you are testing the wrong thing.

If I have one fear of TDD, it is that it can lead to tunnel vision, unless there is someone or something within the project that has the authority and brief to take a much wider overview of the project, and how it will be used.

[Aristotle]

I know what you meant. What I’m saying is that this doesn’t strike me as a danger of TDD so much as a danger of not thinking. Whether you don’t think while following the TDD methodology or you don’t think while following any other methodology (including the no-methodology) really makes no difference – beyond determining in which way you end up screwing up.

I suppose that implicit in your descriptions of the “danger” of TDD is that you intend to express “if you don’t think while following TDD, this is what is likely to happen.” I tend not to find such consequences very interesting because I tend to take thinking along as a given and a prerequisite, which makes your elaborations sound like they’re implying something more along the lines of “if you use TDD, you are likely to stop thinking.” Next to your statements showing you understand TDD quite well this creates an odd sort of cognitive dissonance, and that made me comment.

If I’ve read you correctly, then now you also know how to read me.