|P is for Practical|
Testing my tests (mutation testing)by szabgab (Priest)
|on Feb 26, 2017 at 13:52 UTC||Need Help??|
szabgab has asked for the wisdom of the Perl Monks concerning the following question:
Devel::Cover can easily show if a certain function or expression was executed during the test run, but it cannot tell if there was an assertion checking the validity of the result.
One could randomly change the code under test (e.g. replace a + by a - ) and run the tests again. If they still pass, we have a problem. The tests do not check that code properly.Is there a tool for Perl that would automate this process?
For further clarification, I'd like to change the source code of the module or application under test and leave my tests unchanged.
Update: Use CaseTo further elaborate assume you have a huge code-base with a huge test suite. You pick a module that has 100% coverage and wonder can I safely refactor this? Will the test suite catch if I make a mistake? Lanx gave an excellent and very simple example of code with a problematic test
How can I estimate the risk of changing this code? One possible way is to change the code in a way that should break it and see if the tests fail. So I'd have a tool that can introspect the source code of my application and change the code at a random place. e.g. it would change the + in the above function to -. (No mocking, really changing the code on the disk.)
The tests would still pass.
This is an indication that the tests don't protect me at that point.
AnswerApparently this is called "mutation testing" and there are several modules on CPAN that do this:
See details under this node: Re: Testing my tests