Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

The value of test-driven bug reporting

by xdg (Monsignor)
on Oct 14, 2005 at 03:49 UTC ( #500115=perlmeditation: print w/ replies, xml ) Need Help??

As my interest and skills at automated testing in Perl have grown, I've started applying my test-driven development approach to how I diagnose and report bugs in other people's modules. I think a test-driven bug report has several benefits.

First, by phrasing my bug report in the form of a test, I give the original developer a head-start on fixing the bug and confirming the fix. Second, once I'm already working with someone's test files, I'm more confident in exploring possible fixes, as I can quickly try something out and see if it fixes the issue I found without breaking anything else. Whereas previously I might have just reported an issue and waited for a fix, I now find myself taking a quick peek to see if I can actually fix it now.

It's a wonderful feeling to patch a bug you found and watch the rest of the test suite just hum along smoothly. If you're desperate, you can even install that privately fixed version immediately, knowing that it's just as functional as an unpatched version.

Of course, this only works for modules with good coverage and clean style in testing, but the prevalence of good unit tests and tools like Devel::Cover, plus the strong testing culture in the Perl community make it likely that you'll find at least some tests for any major module you encounter. And for module developers, this is a reason to write your tests cleanly and refactor them periodically -- wouldn't you rather receive bug reports in the form of a test file?

Here's a quick checklist for getting started with test-driven bug reporting. It should be within the capabilities of most intermediate Perl programmers, presuming some familiarity with Test::More and other major test modules, as well as distribution build systems like ExtUtils::MakeMaker and Module::Build.

  • Download and unpack the distribution tarball. Hint: with CPANPLUS, use the cpanp command line tool to download it quickly:

    $ cpanp -d --fetchdir=. MODULE
  • If you use a version control system (e.g. Subversion, SVK, etc.), import the downloaded distribution into a private working area so you can track your changes and easily compare them to the original to produce patch files.

  • Write a test file or patch a test file to demonstrate the bug -- include any additional details in comments in the test file.

  • If you feel confident, look through the code to see if you see a possible fix. If you don't know where to start, try stepping through the test file in the debugger and watch how your test executes.

  • If you find a fix, try it out against the full test suite and see if all the tests pass. If so, great! If not, make a couple more attempts if you can invest the time in it.

  • Post your tests (and patch, if you wrote one) to RT, to a mailing list for the module, or to the author's CPAN address (or to all three, plus any other support forum that may exist).

Of course, it's not always possible to figure out someone's testing approach or to replicate a complicated bug quickly in the time you're willing to allot to it. However, if you can, I think you'll find it's a much more satisfying contribution than posting a simple complaint. And if you can only send a test and not a patch, I still think you're more likely to see a quicker fix to your bug.

Happy testing,

-xdg

Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Comment on The value of test-driven bug reporting
Select or Download Code
Re: The value of test-driven bug reporting
by mirod (Canon) on Oct 14, 2005 at 05:18 UTC

    Yes, yes and yes! For a module author it is so much easier to get a proper test: I can just run the code and figure out what happens, even if the (text) description of the problem is not quite clear to me. I can also integrate the test in the existing test suite easily. All Good Things™ !

Re: The value of test-driven bug reporting
by revdiablo (Prior) on Oct 14, 2005 at 14:52 UTC

    I think this is a great approach. It's interesting to note that they're pushing the same thing with Pugs, and it seems to be working very nicely. The catch phrase bantered around is usually something along the lines of, "I'll trade you a bug fix for a test."

    Not only does it give a way to verify the bug fix, but it forces a bit more effort on the part of the reporter. It's not enough effort to be prohibitive (in my experience, anyway), yet it's just enough to raise the quality of the bug reports. All around, it seems like a really nice idea.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://500115]
Front-paged by TStanley
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others imbibing at the Monastery: (10)
As of 2014-07-28 07:49 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    My favorite superfluous repetitious redundant duplicative phrase is:









    Results (193 votes), past polls