Quiet test runner

[adrianh]

Category:

testing

Description:

Quick and dirty test runner for perl that only reports test failures and unexpectedly succeeding todo tests.

Update: typo fixed, thanks tlm.

Update: fixed is_expected_test_result, thanks tomhukins

#!/usr/bin/perl

use strict;
use warnings;
use Test::Harness::Straps;
use IPC::Run3;

my $strap = Test::Harness::Straps->new();

show_failures( $_ ) for @ARGV;

sub show_failures {
    my $file = shift;
    my $result = test_result( $file );
    show_details( $result ) unless is_expected( $result );
}

sub test_result {
    my $file = shift;
    return { 
        file => $file,
        $strap->analyze( $file, collect_stderr_and_stdout( $file ) )
    };
}

sub is_expected {
    my $result = shift;
    return $result->{ passing } && $result->{ bonus } == 0;
};

sub show_details {
    my $result = shift;
    print "$result->{file}\n", 
        map { "* $_->{name}\n" . diagnostics( $_ ) } 
            grep { ! is_expected_test_result( $_ ) } @{ $result->{ det
+ails } };
}

sub is_expected_test_result {
    my $test_result = shift;
    return $test_result->{ok} &&
        ! is_unexpected_todo_success( $test_result ) &&
        ! diagnostics( $_ );
}

sub diagnostics {
    my $test_result = shift;
    return $test_result->{ diagnostics } if $test_result->{ diagnostic
+s };
    return "    unexpected success from TODO test\n"
        if is_unexpected_todo_success( $test_result );
    return '';
}

sub is_unexpected_todo_success {
    my $test_result = shift;
    return $test_result->{type} eq "todo" && $test_result->{actual_ok}
}

sub collect_stderr_and_stdout {
    my $file = shift;
    my $command = [ $strap->_command, $strap->_switches( $file ) || ()
+, $file ];
    my $output = [];
    run3( $command, \undef, $output, $output ) || die "run3: $!\n";
    return $output;
}

[tomhukins]

I find this really useful, thanks. I'm using it as a consequence of the sitation I described in Reporting test failures within cron.

I noticed it doesn't work quite right if I run an incorrect number of tests. For example:

use Test::More tests => 3;

ok 1, 'Happy';
ok 1, 'Cheerful';

We expect three tests but only run two, which both succeed. Fortunately, is_expected deals with this, but unfortunately show_details doesn't report the failure. The altered is_expected_test_result below fixes this reporting problem:

sub is_expected_test_result {
    my $test_result = shift;
    return $test_result->{ok} &&
        ! is_unexpected_todo_success( $test_result ) &&
        ! diagnostics( $_ );
}

I discovered this when running some Test::WWW::Mechanize tests against a server that stops responding, causing later tests not to run.

[adrianh]

Well spotted. Thanks.