http://www.perlmonks.org?node_id=355203

I've been given the job of "improving the quality" of over a million lines of old code (mostly C++, but some Perl and Java too). Most of this code currently has no tests and was not designed with testability in mind. After reading these nodes:

I've come up with a checklist of possible actions. Please let me know if I've missed important ideas or strategies and which of the points below you consider to be the most important. In particular, I'm interested to know the best way to add tests to existing code.

1. Add new tests whenever you find a bug: i) run test (should fail); ii) fix bug; iii) rerun test (should now pass). That is, grow a regression test suite over time.

2. Risk. Find code "hot spots" by talking to people and examining the bug archives. Add tests for hot spots and areas known to be complex, fragile or risky. Refactor where appropriate (first writing a regression test suite).

3. Add new tests based on what is important to the customer.

4. Documentation-driven tests. Go through the user manual and write a test for each example given there. This also improves the quality of the documentation and hopefully focuses on the customer view of the system.

5. Where you fear a bug. A test that never finds a bug is poor value.

6. Write tests for the more "stable" interfaces. If an interface changes often, you must keep changing the test suite in step.

7. Boundary conditions.

8. Stress tests.

9. Big ones, but not too big. A test that takes too long to run is a barrier to running it often.

10. Use code coverage tools to guide where to write tests.

11. Write some mock objects, where appropriate, to reduce dependencies, make tests more predictable, simulate errors and so on.

12. Do some manual code reviews.

13. Do some code reviews using tools, such as static (e.g. lint) and dynamic (e.g. valgrind) code analysers.

14. Don't let the enormity of the task get you down! Incomplete is better than nothing at all.

janitored by ybiC: Change hardcode www.perlmonks.org/index.pl?node_id=nnnnn links to [id://nnnnn], to avoid logging out monks with cookie set to other perlmonks domain (ie: .com or .net, or sans leading-www)

Replies are listed 'Best First'.
Re: What is the best way to add tests to existing code?
by toma (Vicar) on May 21, 2004 at 08:41 UTC
    The approach that I have seen with this much code is to start with statistical tools that estimate the number of defects in various subsystems. Test and fix some sections of the code to see if the estimated defects are really there. Refine your estimates by using the results of the fixes.

    Figure out some sort of reasonable quality goal that uses the same measures as your estimates. Then develop an estimate of the effort needed to reach the goal. Share your estimates with others to see if you are on the right track. Agree on a plan to improve the quality, and execute it.

    Testing this much code requires a good build system to automate compilation and the running of the tests. Without this infrastructure, the other test tactics that you have listed will be difficult to implement in a reasonable way. A daily build and smoke test is a great way to evaluate the system quality.

    Version control needs to be working, also. If the code changes are not measured it will be difficult to estimate the code quality and to focus the effort.

    In a large system like this, once the build is automated, the versions are controlled, and the quality estimates are done, there are usually one or two subsystems that obviously have the most problems. These are your hot spots. Much of the time the hot spots are already known, but sometimes there are surprises, such as when a problematic low-level data store causes what appear to be UI problems.

    I like the various books by Steve McConnell which touch on this topic. There are other worthwhile books that I'm sure others will recommend.

    It should work perfectly the first time! - toma
Re: What is the best way to add tests to existing code?
by dragonchild (Archbishop) on May 21, 2004 at 14:15 UTC
    First place I'd start would be to write a test for every bug in your bug database, closed or not. For one thing, you guarantee that the bug that was fixed 4 years ago is still fixed. The other reason is that you start to write tests for every subsytem that's being used. (If it isn't being used, then who cares if it has bugs or not?) This gives you ideas for tests that aren't specifically for bugs in your bug tracker. Maybe the bug is a specific instance of a general case and the fix didn't fix the general case.

    Other than that, you pretty much have the list down.

    ------
    We are the carpenters and bricklayers of the Information Age.

    Then there are Damian modules.... *sigh* ... that's not about being less-lazy -- that's about being on some really good drugs -- you know, there is no spoon. - flyingmoose

    I shouldn't have to say this, but any code, unless otherwise stated, is untested

Re: What is the best way to add tests to existing code?
by chromatic (Archbishop) on May 21, 2004 at 15:45 UTC

    Your points 1 through 5 are my top five too. I'd add a point 0, though. "Use test-driven development for all new features."

    If you don't do that, you'll fall further and further behind. Fortunately, many new features — in mature applications — touch and refine existing code.

Re: What is the best way to add tests to existing code?
by tachyon (Chancellor) on May 21, 2004 at 09:41 UTC
Re: What is the best way to add tests to existing code?
by adrianh (Chancellor) on May 22, 2004 at 14:23 UTC
    Please let me know if I've missed important ideas or strategies and which of the points below you consider to be the most important.

    I think you're list pretty much has it covered, and I completely agree with chromatic's point about writing tests for new functionality.

    The other strategy that I'd add is making sure that you write tests around any code that you're going to improve.

    If you've got some gnarly code that you want to refactor write some tests to exercise the behaviour that you want to preserve before you change the code.

    In particular, I'm interested to know the best way to add tests to existing code.

    The advice I would give is to remember that tests exist to help improve the code. They're not an end in themselves.

    A counter productive practice that I've seen is to go through a large piece of legacy code and add developer tests for everything. Doing this with legacy code not driven by tests will produce a test suite that is brittle in the face of change. When you get to the refactoring you're going to find that you're going to be continually throwing away a lot of the new tests so you don't get any benefit from them.

    In my experience it's much more effective to build the test suite around the changes you make to the code. Add tests when you add new features. Add tests around code that you're refactoring. Add tests to demonstrate bugs. In my experience just following those three rules naturallys build a test suite around the most important code.

Re: What is the best way to add tests to existing code?
by qq (Hermit) on May 21, 2004 at 23:22 UTC