Beefy Boxes and Bandwidth Generously Provided by pair Networks
Syntactic Confectionery Delight
 
PerlMonks  

Comment on

( #3333=superdoc: print w/ replies, xml ) Need Help??
Regression testing is important. When writing Perl modules with ExtUtils::MakeMaker, it's usually quite simple, as well. Personally, I often find writing these tests to be both fun and reassuring.

However, like many people, I'm guilty of not writing tests for Every Single Module I write or work on. Often when I skip writing tests, it is because the module I'm working on has external resource dependencies, which I don't think the tests can always depend on, and/or I don't want them to use for testing purposes. At best, this makes writing tests less than simple. At worst, it can make writing tests more expensive than just rewriting the module later when it's found to be broken.

Some examples of resources a module may depend on are (by "internal" I mean things the module programmers have direct control over; external things are outsourced or developed out-of-house):

  • database servers
  • other internal servers/services
  • other internal modules with resource dependencies your module shouldn't need to care about
  • external modules with unknown dependencies
  • external servers/services (Credit card fulfillment, for example)
  • firewall status on the server your tests are running on
  • environment: must be running under mod_perl/apache/etc. to work
We need to test the portions of the module which depend on these resources (it might be every part of the module), or the module isn't tested properly. But we can't always use the same resources for testing as for production, which also makes testing incomplete.

Making the assumption that your test scripts always have access to these resources can be fatal:

  • the resources may be unavailable to the test script/server, causing the tests to fail. Sometimes this means a resource needs to be installed; sometimes it means we've firewalled it to protect it from test scripts :)
  • the resources may be available, but set up in production mode, and tests may break the production data
  • externally controlled resources may not have a test mode available
In these conditions, what do people typically do to build useful test suites? I'm specifically thinking of the case where it would be impossible or impractical to simulate every resource dependency (such as when the amount of code in the new module is small, but the level of resource dependency is high: you'd spend more time simulating external resources than in writing or testing the code).

I'm also interested in techniques people use to ensure that only test resources are used while tests are being performed; especially when using a (possibly externally controlled) module with unknown resource dependencies.

Some solutions we've used, but which haven't provided a complete answer yet, are:

  • test databases with a different DSN in the test script
  • propagate generic "test mode" flags to modules with known dependencies. This isn't always doable for external modules.
  • don't write tests when the test is known to be destructive
  • use a "todo" test when it is likely to fail
  • comment tests which may fail due to known resource unavailability
Any other ideas? I may have thought too much and mostly answered my own question, but I'm still interested in reading how others have dealt with this sort of problem in the past. Thanks!

Alan


In reply to Regression testing with dependencies by ferrency

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post; it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • Outside of code tags, you may need to use entities for some characters:
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.
  • Log In?
    Username:
    Password:

    What's my password?
    Create A New User
    Chatterbox?
    and the web crawler heard nothing...

    How do I use this? | Other CB clients
    Other Users?
    Others musing on the Monastery: (8)
    As of 2014-08-20 11:59 GMT
    Sections?
    Information?
    Find Nodes?
    Leftovers?
      Voting Booth?

      The best computer themed movie is:











      Results (112 votes), past polls