|more useful options|
Re^9: Self Testing Modulesby demerphq (Chancellor)
|on Dec 20, 2005 at 09:13 UTC||Need Help??|
[Assertion failed] would be a lot more useful than [Test Failed]
I think this comment indicates that you don't have the same expectations of a test framework as I do. When I use a test framework I'm testing behaviour, such as the output of a function under certain circumstances. I'm not concerned with implementation im concerned with output and side effects. It really doesn't matter to me if Module.pm(257) expected to get a SCALAR ref and got something different, what matters to me is how that affected the output, or whether it prevented something from happening that should have happened.
As an example in DDS i have assertions in various places, as well as "insane case" handlers on the important branch points. However, knowing that one of these assertions failed doesnt help me nearly as much as knowing what input set lead to the assertion failing. Because of the structure of DDS I can end up executing those assertions on almost any input, so knowing that the assertion failed simply tells me that "for some unknown input something doesnt do the right thing". This doesnt help me nearly as much as knowing that "for a stated input something doesn't do the right thing". So when a test reports a failure I can easily identify the input code responsible, and then work towards the logic failure that caused it. Simply knowing that a logic failure occured doesnt help me.
When I download a module and it fails test the first thing i do is open up the test file to see what was happening then i look at the relevent code in the module and usually I can find a fix right away, with the definition of "fix" being that i can patch it so it passes test. I then continue on my way, satisifed that I know whats going on. On the other hand when I install a module and it fails I have to take considerable time to figure out why because i dont know if its me doing something wrong or if its the module doing something wrong, furthermore since I dont know exactly what it should be doing its much harder to fix it.
So I guess to me in simple terms regression tests are about why the code fails not so much what code failed. Or in other words its about identifying initial states that lead to failure, not about the exact internal cause of it.