Beefy Boxes and Bandwidth Generously Provided by pair Networks
Don't ask to ask, just ask

Skip tests if/unless $ENV{FOO} is set using Test::More

by merlyn (Sage)
on Apr 29, 2003 at 16:53 UTC ( #254011=perlmeditation: print w/replies, xml ) Need Help??

There are a few CPAN modules that have excessive testing time when being installed (or reinstalled). Hint: if it takes more than a minute or two, I'm likely to be irritated.

In class today, it occurred to me that these tests could be conditional upon the presence or absence of an environment variable:

use Test::More tests => 120; ... SKIP: { skip "skipping really long tests", 95 unless $ENV{ALL_TESTS}; ... 95 tests here ... }
That way, the developer can capture the full test suite for development and maintenance (make test ALL_TESTS=1), but as an installer, I don't have to sit through the whole thing, and I can know just how little I can get away with for a first-order "ok to install" test.

Please consider this if you're a module author. Be kind to your remote installer person.

-- Randal L. Schwartz, Perl hacker
Be sure to read my standard disclaimer if this is a reply.

Replies are listed 'Best First'.
Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by Rex(Wrecks) (Curate) on Apr 29, 2003 at 20:55 UTC
    IMHO (and I have made software and hardware QA my business :) it looks like the problem here is that the test scope is pointed in the wrong direction.

    I would hope that all of these tests are being run on as many platforms as are available BEFORE the code is ever downloaded. It seems to me that these tests should be a small, basic subset of the major test suites, and...

    ...Here's the kicker...

    ...should be testing that the module has INSTALLED properly, and the basics work as expected on that platform!

    22,000 tests is not the basics, it is rather excessive for testing if the installation was successful. Having an option to run all 22,000 tests? Sure, someone might need to know the results of that for various reasons, and if they have the requirement to get that knowlege they will hopefully have the patience to get it.

    I would really encourage people to ship as many of their tests as possible with the modules and packages, and even have instructions on how to run them, the debugging benifits are obvious. But please not as an install test, an install test should test exactly that, was the installation successful?

    Scope is a very important thing to keep in mind when applying testing to anything. Just because you can and do execute these tests, does not mean they are appropriate for every place that testing is required.

    How would you feel if every test ever written for your OS was executed every time the computer booted? or even every time you did an install?

    "Nothing is sure but death and taxes" I say combine the two and its death to all taxes!

      I don't think this approach will work well for Perl/CPAN. It's not practical for the average CPAN contributer to test on all perl platforms, perl versions and possible prequisite module versions before distribution.

      Stick a module on CPAN and you get a distributed testing farm for free, with CPAN Testers thrown in as a bonus.

      It's one of the hidden benefits of CPAN and one of the reasons Perl modules end up being so robust (IMHO :-)

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by adrianh (Chancellor) on Apr 29, 2003 at 19:01 UTC

    I have to admit this is something I'd never do. If there's a test I want it run, otherwise I wouldn't have written it. I'd much rather have somebody wait a few minutes than have a module installed with test failures. That way lies madness :-)

    There's also the overhead of printing the test output which schwern has said he'll fix at some point. For modules like M::BI that may be significant.

      Hear hear!

      MJD says you can't just make shit up and expect the computer to know what you mean, retardo!
      I run a Win32 PPM repository for perl 5.6x+5.8x. I take requests.
      ** The Third rule of perl club is a statement of fact: pod is sexy.

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by TVSET (Chaplain) on Apr 29, 2003 at 17:50 UTC
    In class today, it occurred to me that these tests could be conditional upon the presence or absence of an environment variable:

    Are you saying that you were not ready for the class with all installed modules? :)

    On a serious tone: Not going through all tests might reveal mistical problems and failures afterwards. Especially, if you install a module as a dependancy for another module. Sometimes things can get really deeply interconnected (thing HTML::Mason).

    I am thinking more in the ways of My::Module::Simple, which would have less functionality, and therefor less dependencies and tests. It could also give a preview of what My::Module is actually doing and if I need to go through all the hassle of the full-featured installation.

    Leonid Mamtchenkov

      I'm saying that the 96 CPU seconds that Math::BigInt spends running 22,000 tests (yes, literally 22 thousand tests) is not necessary for each upgrade. I'd be satisfied with about 100 key tests, and the rest could be set aside for masochists.

      I mean c'mon, testing is cool and all that, but do you really need to test 22,000 points in the whole spectrum?

      -- Randal L. Schwartz, Perl hacker
      Be sure to read my standard disclaimer if this is a reply.

        Now that you've mentioned 22,000 tests, I tend to agree more with you. :)

        Leonid Mamtchenkov

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by halley (Prior) on Apr 29, 2003 at 17:15 UTC
    This sounds great, but the "accounting" in your example looks like a hassle. Maybe there's a good way to do the following (pseudocode):
      use Test::More tests => 25; # 25 unconditionally ... ADDITIONAL: { skip "Skipping additional tests; try ENV{ALL_TESTS}\n", ADDITIONAL unless $ENV{ALL_TESTS}; $Test::More::tests += 95; # 95 more tests here ... }

    [ e d @ h a l l e y . c c ]

      1. To my eyes this looks more complex than merlyn's example.
      2. You can't do it this way since you need to know the total number of tests before the tests are run so Test::Builder can output an appropriate test header.

        If you're using Test::Harness of 2.0 or newer, you don't need to set a test header. It's still a good idea in cases where you can accidentally run too many tests (see the tests for File::Find, if you're lacking essential horror in your life), but it's not required.

        schwern's even suggested relaxing the requirement for headers in general.

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by lachoy (Parson) on Apr 29, 2003 at 23:37 UTC
    This is pretty timely for me -- I've recently been setting up an environment and writing tests for the next version of OpenInteract. Most of the tests are fairly simple, but they just don't work outside of a running application environment. And it takes a little while to setup that environment -- about 10-20 totally unoptimized seconds right now.

    I guess I had the unrecognized assumption that running the full test suite was a good thing and most people -- that is, the default for whatever tests I write -- wouldn't mind running them.

    I'll have to mull this over for a while... I see the point but I'm also greedy with the mass of testbeds out there available to test on. Like adrianh said, it's one of the powerful (but underrecognized) benefits of CPAN and the infrastructure built up around it.

    M-x auto-bs-mode

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by bronto (Priest) on Apr 30, 2003 at 09:51 UTC

    I just used a similar trick for a module I am going to release (hopefully) soon. To test it extensively I need a directory server handy, and a subtree in it to try a bunch of operations on it. Obviously I can't assume that the condition applies on every possible site the module will be installed in, so I put this code on the script headers:

    use Test::More tests => 18 ; #18 my $fulltest = 18 ; my $shorttest = 2 ; BEGIN { use_ok('Net::LDAP::Simple') ; use_ok('Net::LDAP::Entry') ; } SKIP: { skip "doing local tests only",$fulltest-$shorttest unless $ENV{TEST_HOST} ; my $server = $ENV{TEST_HOST} || 'localhost' ; my $port = $ENV{TEST_PORT} || 389 ; my $base = $ENV{TEST_BASE} || 'ou=simple,o=test' ; my $binddn = $ENV{TEST_BINDDN} || 'cn=admin,o=test' ; my $bindpw = $ENV{TEST_BINDPW} || 'secret' ;

    So, to run all the tests one should run them at least with something like TEST_HOST='' make test


    The very nature of Perl to be like natural language--inconsistant and full of dwim and special cases--makes it impossible to know it all without simply memorizing the documentation (which is not complete or totally correct anyway).
    --John M. Dlugosz

      Another interesting concept is how Kate Pugh does the test setup with CGI::Wiki - a module CGI::Wiki::TestConfig is created and installed with database server credentials by asking the user some questions, unless it already exists. This allows to have persistent test data even across installs and makes testing the stuff where a server is needed even more automatable.

      Another method I found for testing CGI scripts was to set up a mock CGI object (via Test::MockObject) and to simulate my requests to my script through that, but this strategy won't help you much, as you will want to check that your LDAP database was indeed modified as you planned.

      Personally I also want all tests to be run by default. If they take too long for someone, they should turn to the README on how to cut down testing or simply skip testing at all - if you believe the module works, there is no need for testing.

      perl -MHTTP::Daemon -MHTTP::Response -MLWP::Simple -e ' ; # The $d = new HTTP::Daemon and fork and getprint $d->url and exit;#spider ($c = $d->accept())->get_request(); $c->send_response( new #in the HTTP::Response(200,$_,$_,qq(Just another Perl hacker\n))); ' # web
Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by Abigail-II (Bishop) on Apr 30, 2003 at 23:06 UTC
    Well, Regexp::Common has over 150,000 tests, and I don't plan on lowering that number. If you don't want to run (all) tests, you can always ^C a make test and just do make install. Or skip the make test entirely. Or only run those test files you are interested in.

    After all, what should the cut-off point be? 10% of the tests? That still is 15,000 tests, perhaps that's still too much for some people, while others find it not enough. Reduce it to 1%? That would be ridicilously low. Besides, I already spend over 90% of my time working in Regexp::Common in developing tests, I really ain't looking forward to deciding which tests are important, and which aren't.


Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by tbone1 (Monsignor) on Apr 30, 2003 at 12:36 UTC
    This is an excellent idea in general, not just in CPAN modules. In the places I've worked, we've often had QA people who didn't understand the first thing about the operating system they were on. I've often had to build in 'testing' conditions into the code just so the QA people can test their code. I know this sounds bad, because they would be testing against the test versions of code and databases instead of the production versions; however, getting them to even understand that concept wasn't easy. The idea of them actually remembering to type something like ". ~/.project_test.ksh" was laughable.

    (Please note that not all QA people with whom I've worked have been this way, but more than a few have; I'm sure we could all swap stories. If the slumping economy has had one positive effect, it's that we can now pick and choose better QA people than in the last few years.)

    Anyway, I often provide "-P" and "-T" options (where applicable) to force things to production and test, respectively. It saves time, as merlyn noted, and forces one to consider development/test/production issues early, rather than later when they would need to be shoe-horned in awkwardly.

    Ain't enough 'O's in 'stoopid' to describe that guy.
    - Dave "the King" Wilson

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by Anonymous Monk on Apr 29, 2003 at 20:09 UTC
    This only helps if everyone uses the same convention. Which means that it needs to be implemented in one important place. Such as within Test::More itself. For instance you can have an environment variable which, if it is set, will run the first 30 tests and then only tests thereafter which are specifically marked as truly being important.

    If Schwern objects, tell him that testing is fine and dandy, but you have to walk a practical line between the benefit of running tests, and people's willingness to go through it. Right now a lot of modules are going way over...

      Why does everyone need to use the same convention? The only requirement is that the default condition is to skip the tests. If you need to run all of the tests, you should probably look in the README file that tells you how to run all of the tests.

Re: Skip tests if/unless $ENV{FOO} is set using Test::More
by Jenda (Abbot) on Apr 30, 2003 at 17:12 UTC

    There is another problem with tests. Quite often to make a meaningfull test you have to ask the user some questions or ask him to do something. Eg. since most of the functions in Win32::FileOp are supposed to create some kind of dialogs I have to ask the user to click the dialogs off (OK, I would not have to. I could require that they have Win32::Setupsup installed and hope they will not interfere with the dialogs appearing and disappearing on their screen.

    But what if the test is run by some automated tool. Eg. by the one ActiveStates use for their PPM repository. How do I as a module author?

    Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.
       -- Rick Osborne

    Edit by castaway: Closed small tag in signature

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://254011]
Front-paged by broquaint
[jedikaiti]: 'ello, Monklings
[karlgoethebier]: hello

How do I use this? | Other CB clients
Other Users?
Others romping around the Monastery: (5)
As of 2017-06-28 18:01 GMT
Find Nodes?
    Voting Booth?
    How many monitors do you use while coding?

    Results (642 votes). Check out past polls.