Refactoring makes unit tests obsolete

[FoxtrotUniform]

There's something that's always bothered me about unit tests. Basically, it's because I'm ridiculously pedantic, and it grates on my nerves when people say things like:

"If your refactored code passes all of its unit tests, then you haven't broken anything."

That's not true: if your refactored code passes all of its unit tests, then you haven't broken anything in the ways that you've thought to test for. Most of my code has to pass unit and integration tests (the bits that don't are difficult to test; real-time graphics stuff, mostly), but every time I change my code I get a little itch in the back of my brain that says, "What if you've introduced a bug that your original tests didn't check for?"

I don't think this is an unreasonable what-if. For instance, I might write a simple algorithm to deal with a data set that I don't fully understand. With more experience, I understand the data better and write a more complex algorithm to deal with it. Writing that more complex algorithm, I introduce new corner cases that I hadn't considered beforehand, and I probably introduce a bug at one of them. The code passes the unit tests fine -- those unit tests don't check the new corner cases -- and the first I see of the new bug is when my code segfaults in front of a visiting grant-agency representative.

Seems like the answer is to refactor (or at least reconsider) the tests every time you refactor the code. That sounds at first like a heavyweight solution, but it might just be my natural laziness coming to the fore. How do you folks deal with this issue?

[tye]

One method (or goal) of UT is to cover ever path of execution (where you can interpret that as 'every combination' or 'every line').

You can sometimes do quite a bit of refactoring without changing the basic decision points and your UT remains as perfect a fit as before.

You can also drastically refactor the code but keep the basic purpose intact. Much of your old UT should still pass (or can be adapted to take into account rather peripheral changes such as API adjustments but still be a useful set of tests).

Sometimes the old UT becomes mostly useless.

But you usually don't refactor the entire system at once so you'll have API contracts with other parts above and below and the UT should stress the bounds of these contracts.

So, at the least, your old UT should give you a good indication of whether the old contracts are still appropriate and how much you'll need to modify the consumers of the refactored code in order to use it.

[BrowserUk]

Isn't that the point in unit tests?

When a bug turns up it forces you to consider if it is: A) an introduced bug not present in the original, or b) a bug you hadn't tested for originally. If the former, you correct the code, if the latter, you add a new test for it.

As the process iterates, the test suite gets stronger and the possibility of bugs getting into the field diminish.

[DrHyde]

Try not to think of the tests as testing the code. Think of them as testing your documentation. Good documentation will describe all the different types of input to your program (or to each function your library exposes, or whatever) - those types might be FOO, BAR, BAZ and a catch-all "anything else" - and all the possible types of output. Possible outputs include things like a calculated value, a list of values, "dunno", "i've died because your data was stupid" and so on.

As an example, imagine a function is_even(). You might document it as "this function takes a positive integer, and returns 1 if it is even, 0 if it is odd. In all other circumstances it die()s with the message "your father lies down with sheep".

Write your tests to make sure that all of what you've documented actually happens. So there's several things to test here:

• does it die when given values 0, -1, 1.3, "weasel", "4 bees" or a reference?
• does it die when you pass it a list (remember, it takes *a* positive integer)?
• does it return 1 for each of 2, 1000000, and 2**189?
• does it return 0 for each of 1, 999, and 2**3000 - 1?
• does it emit a warning for any of the above tests?
Yes, my choice of some of those data come from experience of how stuff fails in perl and other languages :-)

In your example of rewriting stuff because you understand the problem better, presumably this means that you will have updated the docs to better describe what it does. So you update your tests *because the documentation has changed*.

OK, that was a bit of a simplification! Obviously real-world tests are influenced by the code you're testing. When you write the code you *know* what its corner-cases are. You can see where (eg) the off-by-one errors might be and what would trigger them. So you would write extra tests for those cases.

[EdwardG]

As much as I like unit tests, and the whole meme of test-driven development, there is a kind of circular argument that goes something like

1. Unit tests catch all errors introduced by refactoring
2. ...except for those error it doesn't catch
3. But you add tests for these errors when you notice them
4. ...therefore (see 1) unit tests catch all errors introduced by refactoring

It reminds me of an infinite regression, a la Zeno's paradox (the one about the tortoise in a footrace).

[Zaxo]

As you find new tests you need, it will be interesting to apply them to the old code. If you're doing well, you're likely to find unsuspected bugs in the old stuff.

[Anonymous Monk]

unit tests are important, but even passing all unit tests doesn't mean your program is correct - even if you did think of everything. The whole is more than the sum of its parts. To make an analogy, to find out whether a house you are going to be is structually safe, it isn't sufficient to test each brick, pillar and beam. If you dismantle a safely build house, and use the parts (who each pass their "unit" test) to build a new house, it's still possible for the new house to collapse.

That's the same with software. While a unit test can show a program has defects, it can't show the absence of all defects (and for more fundamental reasons than "not thinking about certain cases").

[McMahon]

I make a living doing the *other* kind of testing: acceptance tests, system tests, functional tests, whatever you want to call it. (Sometimes erroneously called "QA".)

I don't think that the first purpose of unit tests is to find bugs. I think that the first purpose of unit tests is to provide some assurance to the programmer, the maintainer, and the tester that the code in question is in fact sane. That the code has been exercised in some reasonable and thoughtful fashion before being inflicted on the user (whoever that is).

Put another way: write enough unit tests to prove to yourself and to those who come after you that you really did pay some attention while you were coding. You can't write a test that will catch every bug. Trying to do so is a waste of time.

Put yet another way: as a programmer, your job is the trees; as a tester/QA guy, my job is the forest. We're both walking around in the same woods, but we're paying attention in very different ways.

[jplindstrom]

I remember once when I was about to try out a new way of rounding monetary amounts in a big application. The rounding routine was low level stuff, called from the entire program. Ideally the outside API wouldn't change that much.

So I changed the routine (which wasn't a refactoring, but a minor but actual change in logic) and ran the test suit.

Only two out of maybe twenty sets contained failing tests. Hmmm... is this a good thing or a bad thing?

A little of both. It was a good thing, because my change broke very little of what I had bothered to test. But this also showed I had a blind spot in the tests wrt what I just changed, and had I written tests at a more detailed level, more tests probably would have failed. I had to write more tests to verify what I expected from the program. Without a test suite... eh, I don't even want to think about that.

My story is related to refactoring.

When you refactor, you increase your general knowledge about the code, and in what ways it may need to be further tested. This happens when looking for refactoring opportunities and while changing the code.

So if you know your tests and your code you'll hopefully understand where you may be vulnerable, and you can add tests, before, during and after you refactor to counter that.

/J

[SpanishInquisition]

I'm on the edge of this whole unit test thing. It's not that unit tests are bad, they aren't, but for certain applications ... say systems monitoring and GUI apps (not always Perl), it's incredibly hard to write good unit tests. Small code (i.e. modules) can have unit tests pretty easily, large apps can be a challenge.

Furthermore, unit tests can help find problems, but fully successful unit tests can lead to false positives. You still can't test to see if your code will seg fault on RH QU 3 ... no matter what you do.

Meanwhile, if you are only unit testing your external API's, yes those API's shouldn't break when refactoring things. But if you are testing internal API's, well, it's ok to break those!

[dws]

It's not that unit tests are bad, they aren't, but for certain applications ... say systems monitoring and GUI apps (not always Perl), it's incredibly hard to write good unit tests.

If you think in layers, and test at interface boundaries, even systems monitoring applications become reasonable to unit test. It's a rather limited slice that actually worries about things like SNMP packets on the wire, and these parts can be stubbed using mock objects.

Quite a few people have pushed through the "it's incredibly hard" barrier and lived to tell that it's simpler than they'd feared, even for GUI tests. Look around a bit, and you'll find lots of good ideas for writing simple tests.

[FoxtrotUniform]

Quite a few people have pushed through the "it's incredibly hard" barrier and lived to tell that it's simpler than they'd feared, even for GUI tests. Look around a bit, and you'll find lots of good ideas for writing simple tests.

I find that the bits that are really difficult to test are the ones that have to "look right". For instance, I have some normal-mapping code with fairly well-defined requirements, but I'm interested in the final image on the screen. It's difficult to test that programatically: even if I take a screenshot and do a pixel-by-pixel comparison, what do I compare it to? An older result from the same program (hello, circular argument). Besides, the only real way to make sure that some of the stuff I do is correct is to move the viewpoint around and check that it looks "right". I could do a video capture and compare uncompressed, frames, but then I'd have to write and debug and test that code.

What I do instead isn't perfect, but it's pretty good: I write interactive test programs. When I do a build, I get about half a dozen simple little rotate-the-model popups; if they look "about right" I consider it a qualified pass. It's not pretty, but it's better than no tests at all.

On the other hand, I've been quite amazed at the number of tests I've been able to automate that I thought would be "test-by-hand" problems. (Actually, I have some ideas about testing that normal-mapping code.) One of the neat things about testing through an external library is that you have to understand that library very well to write good tests -- you have to understand what comes out the other side, and why.

[McMahon]

Just for posterity, the current canonical list of testing that seems to absolutely require human beings' attention seems to be:

GUI/usability testing
security/penetration testing
performance testing

and many people think that adding some Exploratory Testing to your project is an excellent idea. (Note: Firefox does not like those links. You might need a different browser to view the pages or to get the .pdf.)
UPDATE: benizi helped fix the links, they should work now.