Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot

Perl CPAN test metadata

by eyepopslikeamosquito (Archbishop)
on Aug 21, 2010 at 06:47 UTC ( [id://856415] : perlmeditation . print w/replies, xml ) Need Help??

I've long disliked controlling the running of Perl CPAN module tests via various ad hoc environment variables, such as AUTOMATED_TESTING and TEST_AUTHOR. Somewhat related is the make disttest action, the xt/ directory for "extra" (typically author) tests, and the t/test_manifest file (see Test::Manifest).

I'm a bit rusty on all this stuff, so I did a bit of googling (see References section below) but was unable to find any definitive reference on these issues.

Just as a CPAN module has metadata (META.yml) how about generalizing things like TEST_AUTHOR and xt/ by inventing a standard set of more comprehensive metadata for CPAN tests? Apart from things like TEST_AUTHOR, there is all sorts of other metadata that could be used to describe tests, such as how long the test takes to run, what platforms are supported, and many more. For this to work well, I suspect you'd need a central authority to define the definitive (closed) set of test metadata names, plus a mechanism for growing these names over time.

A simple way to implement this would be by embedding the test metadata in the .t file itself, for example:

# file t/sometest.t # Here is the metadata for test t/sometest.t =begin perltestmeta LONG_RUNNING=1 RELEASE_TEST=1 AUTHOR_TEST=1 SUPPORTED_PLATFORMS=Unix,Windows UNSUPPORTED_PLATFORMS=VMS # ... other test metadata as required =end perltestmeta =cut use Test::More; # Perl tests follow...
CPAN tools would need to be updated to use this metadata, of course.

I'm interested in getting early feedback on this idea before developing it further if appropriate (presumably by posting to mailing list).


The Oslo Consensus (May 2008)

See The Oslo Consensus for full details.

  • xt/ directory for release and other non-install-time tests (subdirectories optional)
  • Support 'requires => { perl => }' and extend to to all 'requires' types
  • *.PL should generate META_LOCAL.yml with requirements after dynamic configuration

The Lancaster Consensus (April 2013)

See The Lancaster Consensus and The Annotated Lancaster Consensus and for full details.

Historically, AUTOMATED_TESTING has been confusing, used for a number of different purposes:

  1. I don't want the user to interact with this test.
  2. This is a long-running test.
  3. This test depends on an external website (say) and I don't want to stop the user installing if it fails, but I want to see what CPAN smokers experienced.

The Lancaster Consensus clarifies the semantics of AUTOMATED_TESTING and RELEASE_TESTING and adds three new environment variables, making a total of five:


To run module tests after installation, use new target "make test-installed", equivalent to "make test" but without adding blib to @INC.

Update: There is also NO_NETWORK_TESTING which, if set, should prohibit anything relying on internet access (see here).

The Berlin Consensus (2015) and PTS Oslo (2018)


The Module::Install::ExtraTests plugin adds one Module::Install command (extra_tests;) which declares that the test files found in the directory ./xt should be run only in certain instances:

  • ./xt/author - run when the tests are being run in an author's working copy
  • ./xt/smoke - run when the dist is being smoked (AUTOMATED_TESTING=1)
  • ./xt/release - run during "make disttest"
These directories are recursively scanned for *.t files. If any directories or files exist in ./xt that are not recognized, the Makefile.PL will die.

References Added Later

  • Perl Toolchain Documents ( by BooK and NeilB) (Oslo Consensus, Lancaster Consensus, Berlin Consensus, Lyon Compromise from Perl QA Conference)
  • toolchain-site - (github) cpan-common, berlin consensus, lancaster consensus, lyon amendment, lyon compromise, oslo consensus

Updated 29-Aug: Added links for CPAN modules Test::XT, Module::Install::ExtraTests et al. Later: Added more references, including the Oslo Consensus, Lancaster Consensus, Berlin Consensus, PTS Oslo.

Replies are listed 'Best First'.
Re: Perl CPAN test metadata
by bingos (Vicar) on Aug 22, 2010 at 23:09 UTC

    Erm, the AUTOMATED_TESTING environment variable being set is me the CPAN Tester letting you the CPAN author know that the environment that is being tested under is an automated smoke tester (ie. no human present). It has no other utility implied or otherwise

      Yes, I know the purpose of AUTOMATED_TESTING, so my post was not as clear as it should have been. Let me clarify the point I'm trying to make with a specific example. Suppose, as a CPAN author, I've written a long-running stress test, t/stress.t say, for my distribution. With AUTOMATED_TESTING, the test writer is expected to write some (imperative) code in t/stress.t to check for the AUTOMATED_TESTING (and possibly other) environment variable/s and skip the test if this variable is not set.

      Instead of asking the test to check its runtime environment, I'm proposing that the test tool/s check the test metadata. In this example, the t/stress.t test (declaratively) states, via test metadata, that it is a long-running stress test. Armed with this metadata, the CPAN tool chain can hopefully "do the right thing": "make test" run by a human would skip the test, while automated smoke testers would cheerfully run it.

      While the details are yet to be fleshed out, I find this approach attractive for three reasons:

      • Easier for test writers. Declarative trumps imperative. For example, the stress test writer does not need to write any (imperative) code to check for any environment variable/s; instead he/she simply states (declaratively) that this is a long-running stress test. Moreover, as environment variables (such as AUTOMATED_TESTING) change over time (with new ones added or old ones removed), the test scripts must be updated in-step.
      • Environment variables are a generally unattractive mechanism because they are an OS-level (potentially insecure) global variable that may not be available in all environments that Perl may want to run in. (Update: in some environments the global AUTOMATED_TESTING environment variable may already be taken).
      • Using metadata offers interesting extension possibilities in that being a long-running, stress test is just one of many possible test attributes. We may uncover many other useful test attributes if we think about it and once we have a mechanism to describe test metadata.

      Update: If you squint, you'll see that the xt/ sub-directory (for "extra" tests) is just a special case of my more general proposal; that is, placing a test in the xt/ sub-directory states (declaratively) that this test has metadata of being an "extra test" (where I guess "extra test" here means "do not run via 'make test' action").

        I don't understand what advantage this has over xt? Anything extra that only author wants to run you can stuff in xt and run with prove

        I have a few thoughts on this, perhaps unrelated to each other.

        AUTOMATED_TESTING is not just limited to Perl. It is used by many test frameworks to indicate an automated testing environment -- some know how to read other metadata files, some do not.

        Your proposal suggests a method by which the test can communicate to the harness that the test belongs in a certain environment. AUTOMATED_TESTING is a method by which the harness communicates with the test that it is running in a certain environment. Two very different things.

        A single test in a test file (rather than the entire set of tests) may need to be skipped if in an AUTOMATED_TESTING environment. Under your proposal, I would need to split this test (along with any initialization, setup, and other concerns) into another file.


Re: Perl CPAN test metadata
by Anonymous Monk on Aug 28, 2010 at 03:59 UTC