Re: Testing my tests

[stevieb]

If I'm understanding correctly, you're wanting something that can introspect the tests themselves, and make sure that you're asserting on the actual return value of a sub or not. Correct?

I think that may be pretty involved to do, as after some quick thought, you'd need to introspect the test file itself, map a test with the sub in question, and validate whether *all* return paths are asserted against:

# module

sub perform {
    my ($x, $y) = @_;

    if ($x < 10){
        return $x + $y;
    }
    if ($x == 10){
        return $x * $y;
    }
  
    return $x - $y;
}

Then:

    is perform(5, 5),  10, "...";
    is perform(10, 5), 50, "...";
    is perform(20, 5), 15, "...";

So it would almost seem as though you'd need to actually write tests against your tests. If there is an existing solution for these types of things, I'd definitely be interested in knowing about it as well.

[choroba]

Boolean conditions are handled by Devel::Cover , it tells you whether you tested the TRUE and FALSE branches of them.

[stevieb]

Right, but I think what he's asking about is whether the return values themselves have been tested against directly. I just put in the if statements to show that to do what he wants, some serious thought would have to be put into the introspection to ensure that if you could prove that an assertion has been made, it would also have to be confirmed which return path the return came out of.

[Anonymous Monk]

"Right, but I think what he's asking about is whether the return values themselves have been tested against directly."

Don't assume, ASK! But consider that if you test your interfaces then those return values WILL BE IN THE TEST. That's what a test does -- it checks that the return value is what you expected. Coverage shows you which branches have been executed by those tests and which have not.

Testing is simple. People make it complicated.

[BrowserUk]

So it would almost seem as though you'd need to actually write tests against your tests.

Won't he then need some tests to test the tests he uses to test his tests?

[GrandFather]

and then a guard to guard the tests against change.

[stevieb]

I've just always felt that if I have 100% coverage of each sub, I must be testing the return value. To boot, I always overlap tests so that it's often not just one test file that uses the return of a sub, so that if I break that return, I may end up with broken tests in several places, if not just several tests within a single file.

[szabgab]

No. I would like to introspect and change my module or application and then run the unchanged tests again. (Updated the original post too with this clarification.)

[stevieb]

What about running the tests, then modifying the sub, then running again? Here's a quickly thrown together example (in this example I simply mock out the whole function) to see if this is more along the lines of what you're looking for:

package Package; {

    sub perform {
        return $_[0] + $_[1];
    }
}

package main; {
    use Mock::Sub;
    use Test::More;

    tests();

    my $m = Mock::Sub->new;
    my $changed = $m->mock('Package::perform');
    $changed->return_value($_[0] - $_[1]);

    tests();

    done_testing();

    sub tests {
        is Package::perform(5, 5), 10, "5 + 5 = 10";
    } 
}

Output:

ok 1 - 5 + 5 = 10
not ok 2 - 5 + 5 = 10
#   Failed test '5 + 5 = 10'
#   at pack.pl line 24.
#          got: '0'
#     expected: '10'
1..2
# Looks like you failed 1 test of 2.

If this example is more along the lines you're after (modifying code on the fly), my Devel::Examine::Subs is designed to alter code within a file (so say you wanted to modify a single line in a single sub within a package, you could (then revert it back), and I could write a mock up example of what it may look like. But perhaps I'm way off here.