Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation

Bailing out from Test::More after 'some' failures

by talexb (Chancellor)
on May 15, 2019 at 15:33 UTC ( #11100024=perlquestion: print w/replies, xml ) Need Help??

talexb has asked for the wisdom of the Perl Monks concerning the following question:

I'm fixing tests on an old code-base, and have come to realize that it would be really handy to be able to ask Test::More for the number of failing tests. I'd like to use that to stop testing after I see a dozen or so failures (not including failures inside a TODO block).

My methodology is to run a limited number of tests, then go back and figure out how to logically put the failing tests inside a TODO block.

Alex / talexb / Toronto

Thanks PJ. We owe you so much. Groklaw -- RIP -- 2003 to 2013.

  • Comment on Bailing out from Test::More after 'some' failures

Replies are listed 'Best First'.
Re: Bailing out from Test::More after 'some' failures
by Perlbotics (Bishop) on May 15, 2019 at 18:26 UTC

    Quick'n dirty. Uses the fact, that Test::More::builder() is not overridden.

    use strict; use warnings; use Test::More; #-- builder() does not exist in Test::More, just intercept call hierar +chy... sub Test::More::builder { my $limit = $Test::More::BAILOUT_LIMIT; my $tb = Test::Builder::Module::builder(@_); my @tests = $tb->summary; my $passed = grep { $_ } @tests; my $failed = @tests - $passed; if ($failed >= $limit) { $tb->BAIL_OUT("Limit of $limit errors reached (passed=$passed, fai +led=$failed)!"); } return $tb; } #-- here, we set the limit $Test::More::BAILOUT_LIMIT = 2; # or i.e. ... = $ENV{BAILOUT_LIMIT} ok( 1, "okay"); is( 1, 2, "oops"); ok( 0, "test $_/3 will fail..." ) for (1..3); done_testing;


    ok 1 - okay not ok 2 - oops # Failed test 'oops' # at ./ line 29. # got: '1' # expected: '2' not ok 3 - test 1/3 will fail... # Failed test 'test 1/3 will fail...' # at ./ line 30. Bail out! Limit of 2 errors reached (passed=1, failed=2)!

    Can be done properly by deriving your own class, but you get the idea. You need one more test than the limit, though. The interception happens before the next test case is executed. Limit = N demands N+1 tests, but I guess that is acceptable...

    Edit: This implements an individual per-test-file-limit. If you want the limit to encompass the whole test session, you'll need to maintain the counter's persistency and life-cycle across all sessions (i.e. using Storable).

      ++ This is awesome.

      Although I didn't put in much effort at all, I have considered and very lightly attempted to achieve what OP is asking a few times over my 20 years coding Perl.

Re: Bailing out from Test::More after 'some' failures
by exodist (Monk) on May 16, 2019 at 14:33 UTC
    Take a look at Test2::Plugin::BailOnFail. You can probably copy that and change it so that it counts failures and fails after N instead of failing after the first one. Test::Builder is now a compatibility layer around Test2 these days so this will count failures Test::Builder will miss. Also this plugin does not have the problem of loosing diagnostics (most of the time) that the Test::Builder override does that requires the override to wait for the next assertion before exiting. Finally this way of doing it requires no hacks or monkey-patching, instead it uses things put in place to intentionally make what you want possible.
      I'm intrigued by the "most of the time" parenthesis. Could you elaborate?
      map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
        Most of the time means that any tool written with Test2 in mind, or directly on Test2 will work as expected, this should include most methods on Test::Builder itself. However cpan tools written against old Test::Builder that do not use Test2's 'Context' system may still call $tb->ok and $tb->diag separately, in which case the diag will still be lost.

        In Test2 you would do this:
        sub mytool { my $ctx = context(); $ctx->ok(...); $ctx->diag(...); $ctx->release; # This is where the bail-on-fail is triggered }
        The TB version:
        sub mytool { $tb->ok(...); # BailOnFail is triggered here $tb->diag(...); # Lost }
        $tb methods do acquire context's, but each of those methods has to acquire it's own. That said the diagnostics TB itself generates in a failed-ok (like line number) will make the output, it is just extra diags are lost.

        That is unfortunately a fundamental, and not-fixable limitation of Test::Builder that Test2 avoided with the context system. Test2 was specifically designed to overcome this and many other TB limitations.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11100024]
Approved by Aldebaran
Front-paged by Corion
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (4)
As of 2022-11-29 17:32 GMT
Find Nodes?
    Voting Booth?