Beefy Boxes and Bandwidth Generously Provided by pair Networks
XP is just a number
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??

I've long disliked controlling the running of Perl CPAN module tests via various ad hoc environment variables, such as AUTOMATED_TESTING and TEST_AUTHOR. Somewhat related is the make disttest action, the xt/ directory for "extra" (typically author) tests, and the t/test_manifest file (see Test::Manifest).

I'm a bit rusty on all this stuff, so I did a bit of googling (see References section below) but was unable to find any definitive reference on these issues.

Just as a CPAN module has metadata (META.yml) how about generalizing things like TEST_AUTHOR and xt/ by inventing a standard set of more comprehensive metadata for CPAN tests? Apart from things like TEST_AUTHOR, there is all sorts of other metadata that could be used to describe tests, such as how long the test takes to run, what platforms are supported, and many more. For this to work well, I suspect you'd need a central authority to define the definitive (closed) set of test metadata names, plus a mechanism for growing these names over time.

A simple way to implement this would be by embedding the test metadata in the .t file itself, for example:

# file t/sometest.t # Here is the metadata for test t/sometest.t =begin perltestmeta LONG_RUNNING=1 RELEASE_TEST=1 AUTHOR_TEST=1 SUPPORTED_PLATFORMS=Unix,Windows UNSUPPORTED_PLATFORMS=VMS # ... other test metadata as required =end perltestmeta =cut use Test::More; # Perl tests follow...
CPAN tools would need to be updated to use this metadata, of course.

I'm interested in getting early feedback on this idea before developing it further if appropriate (presumably by posting to perl-qa@perl.org mailing list).

References

The Oslo Consensus (May 2008)

  • $ENV{AUTOMATED_TESTING} - not $ENV{PERL_AUTOMATED_TESTING}
  • $ENV{RELEASE_TESTING} - not $ENV{AUTHOR_TESTING} or $ENV{PERL_AUTHOR_TESTING}
  • xt/ directory for release and other non-install-time tests (subdirectories optional)
  • Support 'requires => { perl => 5.xxx }' and extend to to all 'requires' types
  • *.PL should generate META_LOCAL.yml with requirements after dynamic configuration

The Lancaster Consensus (April 2013)

See The Lancaster Consensus and The Annotated Lancaster Consensus for full details.

Historically, AUTOMATED_TESTING has been confusing, used for a number of different purposes:

  1. I don't want the user to interact with this test.
  2. This is a long-running test.
  3. This test depends on an external website (say) and I don't want to stop the user installing if it fails, but I want to see what CPAN smokers experienced.

The Lancaster Consensus clarifies the semantics of AUTOMATED_TESTING and RELEASE_TESTING and adds three new environment variables, making a total of five:

  • AUTOMATED_TESTING
  • NONINTERACTIVE_TESTING
  • EXTENDED_TESTING
  • RELEASE_TESTING
  • AUTHOR_TESTING

To run module tests after installation, use new target "make test-installed", equivalent to "make test" but without adding blib to @INC.

Updated 29-Aug: Added links for CPAN modules Test::XT, Module::Install::ExtraTests et al. 18-Sep: Added more references, including the Oslo Consensus. June-2013: Added Lancaster Consensus.


In reply to Perl CPAN test metadata by eyepopslikeamosquito

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others having an uproarious good time at the Monastery: (12)
    As of 2014-09-19 13:04 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      How do you remember the number of days in each month?











      Results (138 votes), past polls