Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister

Quiet test runner

by adrianh (Chancellor)
on Jul 30, 2005 at 14:39 UTC ( #479609=sourcecode: print w/replies, xml ) Need Help??
Category: testing
Author/Contact Info Adrian Howard

Quick and dirty test runner for perl that only reports test failures and unexpectedly succeeding todo tests.

Update: typo fixed, thanks tlm.

Update: fixed is_expected_test_result, thanks tomhukins


use strict;
use warnings;
use Test::Harness::Straps;
use IPC::Run3;

my $strap = Test::Harness::Straps->new();

show_failures( $_ ) for @ARGV;

sub show_failures {
    my $file = shift;
    my $result = test_result( $file );
    show_details( $result ) unless is_expected( $result );

sub test_result {
    my $file = shift;
    return { 
        file => $file,
        $strap->analyze( $file, collect_stderr_and_stdout( $file ) )

sub is_expected {
    my $result = shift;
    return $result->{ passing } && $result->{ bonus } == 0;

sub show_details {
    my $result = shift;
    print "$result->{file}\n", 
        map { "* $_->{name}\n" . diagnostics( $_ ) } 
            grep { ! is_expected_test_result( $_ ) } @{ $result->{ det
+ails } };

sub is_expected_test_result {
    my $test_result = shift;
    return $test_result->{ok} &&
        ! is_unexpected_todo_success( $test_result ) &&
        ! diagnostics( $_ );

sub diagnostics {
    my $test_result = shift;
    return $test_result->{ diagnostics } if $test_result->{ diagnostic
+s };
    return "    unexpected success from TODO test\n"
        if is_unexpected_todo_success( $test_result );
    return '';

sub is_unexpected_todo_success {
    my $test_result = shift;
    return $test_result->{type} eq "todo" && $test_result->{actual_ok}

sub collect_stderr_and_stdout {
    my $file = shift;
    my $command = [ $strap->_command, $strap->_switches( $file ) || ()
+, $file ];
    my $output = [];
    run3( $command, \undef, $output, $output ) || die "run3: $!\n";
    return $output;
Replies are listed 'Best First'.
Re: Quiet test runner
by tomhukins (Curate) on Sep 21, 2005 at 15:22 UTC

    I find this really useful, thanks. I'm using it as a consequence of the sitation I described in Reporting test failures within cron.

    I noticed it doesn't work quite right if I run an incorrect number of tests. For example:

    use Test::More tests => 3; ok 1, 'Happy'; ok 1, 'Cheerful';

    We expect three tests but only run two, which both succeed. Fortunately, is_expected deals with this, but unfortunately show_details doesn't report the failure. The altered is_expected_test_result below fixes this reporting problem:

    sub is_expected_test_result { my $test_result = shift; return $test_result->{ok} && ! is_unexpected_todo_success( $test_result ) && ! diagnostics( $_ ); }

    I discovered this when running some Test::WWW::Mechanize tests against a server that stops responding, causing later tests not to run.

      Well spotted. Thanks.

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: sourcecode [id://479609]
[stevieb]: I went on my merry way writing a cross-platform, network-aware system that works across Perlbrew and Berrybrew systems and runs unit tests for Perl dists on all installed versions, with the ability to manage *brew commands themselves
[stevieb]: That worked out exceptionally well, as when I started that project, I hadn't delved into hardware development yet.
[stevieb]: found a issue in MetaCPAN::Client though today for my revdep tests. At least I think it's an issue
[stevieb]: I'm working on hiding the fact the software needs MetaCPAN::Client, as I often have a hell of a time installing it. I'm going to change it to let the user know if they do revdep that it needs to be installed, and remove that distribution ..
[stevieb]: ...from being a prereq

How do I use this? | Other CB clients
Other Users?
Others musing on the Monastery: (4)
As of 2017-06-25 23:33 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (572 votes). Check out past polls.