One of the interesting things about Test::Class is that it loads the perl interpreter once and your modules once. Imagine if you have 30 test programs and each loads Catalyst::Runtime and DBIx::Class. While those are great modules, they take a while to load and reloading them 30 times can be painful. If you convert your 30 test programs to 30 test classes, they only get loaded once and you can get a nice performance boost in your test suite.

Recently, I faced this situation with a bunch of tests which were written as YAML files. Each YAML file had a separate, but nearly identical, test program. These tests took 9.6 minutes to run. I thought about how Test::Class solves this performance issue and I wrote a single test program which found all of the YAML tests and ran them at once. It looked something like this:

use Test::More 'no_plan'; use YAMLTest; my $builder = Test::More->builder; foreach my $test (get_tests()) { my $current = $builder->current_test; my $expected = YAMLTest->run($test); is $builder->current_test, $current + $expected, "$test has good t +est count"; }

By running the tests like this, each module was loaded only once and the total test run dropped from 9.6 minuts to 3.6 minutes. That got me to thinking about a very stupid idea.

Could I do this with regular *.t test programs?

It's a hack, has plenty of problems and breaks in a few cases. It definitely improved some test times, but it does require that tests be written in a way which supports this. Who knows? Maybe someone will find it useful. It was certainly fun to write.

Note that I override a Test::Builder to take out the check for more than one plan. Overridding &Test::More::import is probably a better route.

#!/usr/bin/perl use strict; use warnings; use File::Find; use Test::More qw/no_plan/; my @test_files; find( { no_chdir => 1, wanted => sub { push @test_files => $File::Find::name if /\.t +\z/ } }, 't/' ); sub slurp { my $file = shift; open my $fh, '<', $file or die "Cannot read ($file): $!"; return do { local $/; <$fh> }; } sub get_package { my $file = shift; $file =~ s/\W//g; return $file; } my $code = <<'END_CODE'; { no warnings 'redefine'; package Test::Builder; sub plan { my ( $self, $cmd, $arg ) = @_; return unless $cmd; local $Test::Builder::Level = $Test::Builder::Level + 1; if ( $cmd eq 'no_plan' ) { $self->no_plan; } elsif ( $cmd eq 'skip_all' ) { return $self->skip_all($arg); } elsif ( $cmd eq 'tests' ) { if ($arg) { local $Test::Builder::Level = $Test::Builder::Level + +1; return $self->expected_tests($arg); } elsif ( !defined $arg ) { $self->croak("Got an undefined number of tests"); } elsif ( !$arg ) { $self->croak("You said to run 0 tests"); } } else { my @args = grep { defined } ( $cmd, $arg ); $self->croak("plan() doesn't understand @args"); } return 1; } sub no_header { 1 } } END_CODE my @packages; foreach my $file (@test_files) { my $package = get_package($file); my $tests = slurp($file); next if $tests =~ /use\s+Config;/; # Why does this break things? next if $tests =~ /^__(?:DATA|END)__/m; push @packages => [ $package, $file ]; $code .= <<" END_CODE"; package $package; sub handler { $tests; } END_CODE } $code .= <<'END_CODE'; END { no warnings 'redefine'; *Test::Builder::no_header = sub { 0 } } END_CODE eval $code; if ( my $error = $@ ) { my $file = 'dump.t'; open my $fh, '>', $file or die "Cannot open ($file) for writing: $ +!"; print $fh $code; close $fh; BAIL_OUT("Cannot load modules: $error"); } foreach my $package (@packages) { ok $package->[0], ">>>>>>>> testing $package->[0]"; $package->[0]->handler; }

Cheers,
Ovid

New address of my CGI Course.

Replies are listed 'Best First'.
Re: Stupid, but fun, ideas in testing
by xdg (Monsignor) on Nov 24, 2007 at 13:27 UTC

    That's nice if you're optimizing to minimize test runtime. However, I usually optimize to maximize test independence. I want each *.t file to run in its own interpreter from a known starting point.

    That said, I do often look for ways to run similar tests in one main loop where the test data comes from a data structure within the .t file or from outside the .t file in some way.

    How does Test::Class handle the independence issue? Load all the modules and then fork for each class?

    -xdg

    Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

      You could probably easily hack in that separation to Test::Class, but that's not part of its design. By deliberately allowing different tests to run in the same process, you can easily find hidden assumptions about state. Maintaining a reasonable state is what the startup/setup/teardown/shutdown attributes are for. See this thread for Adrian Howard's thoughts on this topic.

      Cheers,
      Ovid

      New address of my CGI Course.

        Maintaining a reasonable state is what the startup/setup/teardown/shutdown attributes are for.

        But teardown doesn't reset %INC and changes to the symbol tables.

        Sometimes, when I refactor code to a separate module, I might wind up taking a function or class method call along with and I might forget to use or require the corresponding module. If I only test that new module after loading the original, I might never notice that the new module doesn't load something. Only testing that new module on its own (assuming that's a valid use case) would pick that up. Admittedly, that's a simplistic example for argument, but I think that it makes the case that Test::Class shouldn't be used for optimization without some good thought into what the tradeoffs are.

        Of course, I maintain weird modules like Sub::Uplevel and Class::InsideOut, so maybe I just tend to be extra careful.

        -xdg

        Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

        By deliberately allowing different tests to run in the same process, you can easily find hidden assumptions about state.

        One good way to shake out dependencies is to run the tests in semi-random order. "Semi" in the case means at least temporarily reproduceable, to allow for debugging. A switch to run the tests in reverse order is a trivially easy way to flush out some problems. Seeding the random number generator with the epoch date (but not the time) before randomizing test order allows for same-day debugging.

      It doesn't on it's own, it runs the tests in whatever test classes you have loaded. So if you have two test classes named App::Test::Foo and App::Test::Bar, then you can run them together...

      use App::Test::Foo; use App::Test::Bar; Test::Class->runtests;

      In this case they will not be run independently, they will be run in the same interpreter, and I don't think the order they get run in (App::Test::Foo first or App::Test::Bar first) is defined.

      If you want them to run independently, you still have to create independent test scripts for them...

      # Foo.t use App::Test::Foo; Test::Class->runtests; # Bar.t use App::Test::Bar; Test::Class->runtests;

      Generally I don't think of Test::Class as a replacement for traditional *.t, but as a helper for them. It's a great help if you have things like unit tests where you have to setup fixtures first and tear them down after the testing, or if you have a bunch of similar classes that need to have their common features tested as well as their own individual unit tests.


      We're not surrounded, we're in a target-rich environment!