Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Testing test function

by sedusedan (Monk)
on Sep 26, 2013 at 01:11 UTC ( #1055765=perlquestion: print w/ replies, xml ) Need Help??
sedusedan has asked for the wisdom of the Perl Monks concerning the following question:

I'm wondering if someone can point me to some simpler examples of testing a Test::Builder-based test function. I glanced at Test::More's tests once a while back and decided not to do it yet because they looked pretty hairy. But I recently got hit with a bug in my test function (passing things which should not have passed).

UPDATE: Perhaps I should've said "proper and simple" instead of just "simpler". I know that one of the straightforward ways to do this was to run the test in a subprocess and capture/parse its TAP output. Somehow I was led to think that Test::Builder->create is the proper way. Am I wrong? Anyway, I'm going to be pragmatic for now.

Comment on Testing test function
Re: Testing test function (testing library test suite)
by Anonymous Monk on Sep 26, 2013 at 01:22 UTC

    I'm confused (i don't understand the question), are you really asking how to test your testing-library-functions?

      Yup, sorry for not being clearer. My module in question is Gen::Test::Rinci::FuncResult. The module actually *generates* test functions, e.g. test_foo() or test_bar(). The test functions basically do a Test::More::subtest() with some assertions like ok(), is(), is_deeply() inside it.
Re: Testing test function
by kcott (Abbot) on Sep 26, 2013 at 02:11 UTC

    G'day sedusedan,

    "I'm wondering if someone can point me to some simpler examples of testing a Test::Builder-based test function."

    You ask for "simpler examples" but you don't indicate simpler than what. The EXAMPLES section of the Test::Builder documentation says: "CPAN can provide the best examples. Test::Simple, Test::More, Test::Exception and Test::Differences all use Test::Builder.". Are you looking for simpler examples than these? There are links to dozens of tests in the Test::Simple MANIFEST. Have you looked at those?

    "I glanced at Test::More's tests once a while back and decided not to do it yet because they looked pretty hairy."

    I'm not sure whether you're referring to the tests documented in Test::More or the tests written in t/More.t (or, possibly, something else). Perhaps start with Test::Tutorial. You can always ask here if there's something specific you're having problems with.

    -- Ken

      I'm referring to Test-Simple's tests (yes, like the file you mentioned, t/More.t). Basically I have this test_foo() function that uses Test::More and does several calls to ok(), is(), is_deeply() and subtest() inside it. I want to test that test_foo() works. Specifically, aside from making sure that test_foo() can pass a test, I need to test that test_foo() can also fail a test. Something like:

      #test_test_foo.t
      #!perl
      use Test::More;
      ok( test_foo(a=>1), 'test_foo() should succeed if given a=1'); #1
      ok(!test_foo(a=>2), 'test_foo() should fail if given a=2'); #2
      done_testing;
      

      That's the tricky part for me, since doing #2 will fail test_test_foo.t.

      BTW, after looking at t/More.t, I also don't see that it tests failures like #2.

        Untested: What about
        ok(! eval { test_foo(a=>2) ; 1}, 'test_foo() should fail for a=2'); is($@, 'invalid argument', 'expected error');
        لսႽ ᥲᥒ⚪⟊Ⴙᘓᖇ Ꮅᘓᖇ⎱ Ⴙᥲ𝇋ƙᘓᖇ
Re: Testing test function
by DrHyde (Prior) on Sep 26, 2013 at 10:54 UTC
    I presume that deep down inside the function you want to test there is a call to Test::More::ok(). If there isn't, there probably should be. You need to mock that function. I do this in the tests for Class::Mockable.
      Are you saying that I need to mock Test::More's ok()? I also use subtest(), is_deeply(), is(), like() BTW, should I mock those too? Why? I'd rather execute the test in a subprocess and parse the TAP instead.
        You could do that, but beware that the TAP standard is slowly evolving. You need to make sure that you can cope when new features are added to it. You may also need to cope with code that doesn't actually use Test::More and just outputs something vaguely TAP-ish. Lots of old code's test suites contain things like this:
        print 'not ' unless($foo == 42); print 'ok '.$test++."\n";
        instead of
        ok($foo == 42);
Re: Testing test function (fork)
by tye (Cardinal) on Sep 26, 2013 at 14:42 UTC

    Write a test script that includes successes and failures but don't name it as a test script in your test suite. Add to your test suite a test that runs the above test script and verifies that it outputs "ok 1" and "not ok 2" or whatever. Simple.

    - tye        

Re: Testing test function
by tobyink (Abbot) on Sep 26, 2013 at 20:09 UTC

    Test::More comes with Test::Builder::Tester for testing testing modules. It's horrible. Just horrible.

    Use Test::Tester instead. Here's an example of using Test::Tester to test the is_deeply function from Test::More, and make sure there isn't some insane bug in is_deeply that nobody's ever noticed...

    use strict; use warnings; use Test::Tester; use Test::More 0.96; subtest "two identical structures" => sub { my (undef, $result1) = run_tests sub { is_deeply( [1,2,3], [1..3] +) }; ok( $result1->{ok}, 'is_deeply passes', ); is( $result1->{diag}, '', 'no unnecessary diagnostics printed', ); done_testing; }; subtest "two different structures" => sub { my (undef, $result2) = run_tests sub { is_deeply( [1,3], [1..3] ) +}; ok( !$result2->{ok}, 'is_deeply fails' ); like( $result2->{diag}, qr{Structures begin differing}, 'expected diagnostics ok' ); done_testing; }; done_testing;

    TL;DR: run_tests runs some tests in a coderef, captures the TAP output, and turns it into a list of hashrefs of test results. You can then use plain old Test::More functions to test those test results.

    use Moops; class Cow :rw { has name => (default => 'Ermintrude') }; say Cow->new->name
      Thanks Toby, Test::Tester is exactly what I want. I'm gonna take your word on Test::Builder::Tester being horrible, besides it's a bit too low-level for my taste.
      I ended up not using Test::Tester because it has problems with subtests (submitted as wishlist to RT), so I currently run the test in a subprocess and parse the TAP output manually.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlquestion [id://1055765]
Approved by kcott
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (9)
As of 2014-09-30 15:06 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (375 votes), past polls