Quieting Test::More

[stevieb]

Test::More is verbose in its output as it should be. That said, when I have say a loop of tests that produces ~100 results, is there a way to flag it to say "don't say anything, except for my debug print statements"?

Being able to do so would allow me to troubleshoot specific areas of my test files without having to continuously manually edit a test file and move around nonsense like:

done_testing();
exit;

...lines.

Using something like perl t/100-test.t > /dev/null works well to display only the failed test results to STDERR, but that buries print statements.

What do you do to either stop your tests from running upon a failed one, or allow your print statements to be printed to STDOUT and bypass everything else?

[kcott]

G'day stevieb,

"... is there a way to flag it to say "don't say anything, except for my debug print statements"?"

You can do this with a combination of diag statements, SKIP: blocks, and a DEBUG flag. Here's a very simple, and highly contrived, example (pm_1194120.t):

use Test::More tests => 2;

SKIP: {
    skip 'Debugging', 0 unless $ENV{PM_1194120_DEBUG};

    diag 'DEBUG MODE!';
    diag 'Debug statement #1 (1 == 1): ', 1 == 1 ? 'TRUE' : 'FALSE';
    diag 'Debug statement #2 (1 == 0): ', 1 == 0 ? 'TRUE' : 'FALSE';
}

SKIP: {
    skip 'Debugging', 2 if $ENV{PM_1194120_DEBUG};

    is(1, 1, 'Test: 1 == 1');
    isnt(1, 0, 'Test: 1 != 0');
}

Without the debug flag being set, here's verbose, non-verbose, and no STDOUT output (timing information elided):

$ export PM_1194120_DEBUG=0
$ prove -v pm_1194120.t
pm_1194120.t .. 
1..2
ok 1 - Test: 1 == 1
ok 2 - Test: 1 != 0
ok
All tests successful.
Files=1, Tests=2,  ...
Result: PASS
$ prove pm_1194120.t
pm_1194120.t .. ok   
All tests successful.
Files=1, Tests=2,  ...
Result: PASS
$ prove pm_1194120.t > /dev/null
$

Here's the same commands, with the debug flag set:

$ export PM_1194120_DEBUG=1
$ prove -v pm_1194120.t
pm_1194120.t .. 
1..2
# DEBUG MODE!
# Debug statement #1 (1 == 1): TRUE
# Debug statement #2 (1 == 0): FALSE
ok 1 # skip Debugging
ok 2 # skip Debugging
ok
All tests successful.
Files=1, Tests=2,  ...
Result: PASS
$ prove pm_1194120.t
pm_1194120.t .. # DEBUG MODE!
# Debug statement #1 (1 == 1): TRUE
# Debug statement #2 (1 == 0): FALSE
pm_1194120.t .. ok   
All tests successful.
Files=1, Tests=2,  ...
Result: PASS
$ prove pm_1194120.t > /dev/null
# DEBUG MODE!
# Debug statement #1 (1 == 1): TRUE
# Debug statement #2 (1 == 0): FALSE
$

While that last one does exactly what you ask (i.e. nothing but debug statement output); consider whether you really want to throw away STDOUT (it's only a few lines and could provide useful feedback).

Obviously, there's a huge number of variations on that theme. You could have more than one debug flag and they certainly don't need to be environmental variables. I've separated the diag statements from the tests, but there could be a mixture; and, not everything needs to be in a SKIP: block.

See also: prove (for a variety of other options, '-b' is one I usually need); and Test::More (in particular, the Diagnostics, Conditional tests, and Test control sections).

[1nickt]

Like the Anonymous One said, (and wasn't this an answer to an earlier question you had about Test::More?): Don't run your tests with perl foo.t. Test::More is meant to be used inside a testing harness such as is provided by prove. use prove.

$ prove -lr t/050-types.t 

t/050-types.t .. ok    
All tests successful.
Files=1, Tests=22,  0 wallclock secs ( 0.02 usr  0.00 sys +  0.07 cusr
+  0.00 csys =  0.09 CPU)
Result: PASS

Also, use note and diag and explain in tests, then the -v option to prove can be used to control your "debug" output as well as test output verbosity.

[haukex]

Is this the kind of thing you're looking for? See also reset_outputs in Test::Builder.

#!/usr/bin/env perl
use warnings;
use strict;
use Test::More tests=>200;

use File::Spec;
Test::More->builder->output(File::Spec->devnull);

is 1,1 for 1..199; # lots of output

is 1,2, 'test failure';
print "print\n";
warn "warn\n";
diag "diag";
note "note";

__END__

1..200
#   Failed test 'test failure'
#   at tst.pl line 11.
#          got: '1'
#     expected: '2'
print
warn
# diag
# Looks like you failed 1 test of 200.

[Anonymous Monk]

Use prove or look at how prove does it internally: Test::Builder, Test::Harness, etc.

[stevieb]

Although not completely, question was a bit rhetorical in hopes for explanations with examples...

[Anonymous Monk]

IMHO

I think the real problem is using Test::More as a debugger

I get how easy it is to start using testing modules for everything

I get wanting document every debugging session , every development milestones,

but doing it in one file is bad habit, as is doing it using testing modules , the modularity of your API suffers if there is no separation between testing and development

the solution is smaller and smaller teeny tiny test files, like bug specific rt-666.t bug-frobnitz.t, even if 80%-99% of each test file are identical save for the input/output

but if you're gonna debug learn the API :) https://metacpan.org/pod/Test::More#Diagnostics

Update: Link fixed by GrandFather

[stevieb]

I don't think you are understanding completely what I'm asking/referencing here.

I do not do many things within a single file. In fact, I have written accepted patches to ExtUtils::MakeMaker documentation that ensures that sub tests and sub directories are understood within the testing framework, to ensure compartmentalized testing is available, and documented.

I write test files that are specific to a small piece of my software, eg t/05-sub_one.t, t/10-sub_two.t. It doesn't matter if the test file has two or 400 tests in it. I want to know how to shut things up so I can see my print statements only.

You can verify if we're on the same page here or not by looking here.

I feel that my APIs flourish by testing them the way I do. I am definitely open to comments in that regard though, if one looks at one of my APIs, then the tests, then responds about them directly.

[stevieb]

404 on the link... this message will (hopefully) self-destruct. I would have /msgd, but, anonymonk and all.