Beefy Boxes and Bandwidth Generously Provided by pair Networks
Problems? Is your data what you think it is?
 
PerlMonks  

Re: When test-driven development just won't do

by Anonymous Monk
on Aug 04, 2005 at 12:18 UTC ( #480791=note: print w/ replies, xml ) Need Help??


in reply to When test-driven development just won't do

There are a lot of things I would like to test, but I've no idea how to write tests for. For instance, how do you write a test that tests your objects (or other datastructures) timely destruct? That is, they get garbage collected at the moment you think they are, and aren't kept alive (consuming more memory you think they do) longer than they are supposed to due to an unexpected reference loop? Sure, you could add a DESTROY function that sets a global flag, but that would be a heavy price to pay.

How do you test your shuffling or die rolling technique is fair and produces results according to your specifications (say, according to a Gaussing distribution), and isn't biased favouring certain outcomes?

A common case where tests pass is with the can_ok $CLASS, $method; test.
That's a test I wouldn't use without a testing $method itself as well - because it's a test which isn't interesting if it passes. It's interesting if it fails, because that you know the method isn't there, but if it's there, and doesn't do what it's supposed to do, its existance isn't useful. So, if I use this test, it would preceede the tests that test $method.


Comment on Re: When test-driven development just won't do
Re^2: When test-driven development just won't do
by tlm (Prior) on Aug 04, 2005 at 12:38 UTC

    For instance, how do you write a test that tests your objects ... timely destruct?

    How about (untested):

    { package CrashDummy; @CrashDummy::ISA = 'MyPrecious'; *ok = \&Test::More::ok; my $counter = 0; sub DESTROY { Test::More::ok( ++$counter == 2, 'DESTROY called' ); $_[ 0 ]->SUPER::DESTROY; } { my $dummy = bless +{}, __PACKAGE__; ok( ++$counter == 1, 'inner scope' ); } ok( ++$counter == 3, 'expected execution order' ); }
    ?

    How do you test your shuffling or die rolling technique is fair and produces results according to your specifications (say, according to a Gaussing distribution), and isn't biased favouring certain outcomes?

    There are a bazillion statistical tests to compare sampled distributions. I'm surprised none of these would meet your needs.

    the lowliest monk

Re^2: When test-driven development just won't do
by xdg (Monsignor) on Aug 04, 2005 at 13:53 UTC

    For random number generation:

    For garbage collection, if you really want to do this, why not install your own UNIVERSAL::DESTROY subroutine during testing:

    use strict; use warnings; package Foo; sub new { return bless {} } package main; use Test::More tests => 3; my %destroyed; $destroyed{Foo} = 0; { no strict 'refs'; *{"UNIVERSAL::DESTROY"} = sub { $destroyed{ref(shift)}++ }; } my $obj1 = Foo->new; my $obj2 = Foo->new; is( $destroyed{Foo}, 0, "Nothing destroyed yet" ); $obj1 = undef; is( $destroyed{Foo}, 1, "Destroyed 1" ); $obj2 = undef; is( $destroyed{Foo}, 2, "Destroyed 2" ); __END__ 1..3 ok 1 - Nothing destroyed yet ok 2 - Destroyed 1 ok 3 - Destroyed 2

    -xdg

    Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

Re^2: When test-driven development just won't do
by itub (Priest) on Aug 04, 2005 at 13:53 UTC
    I used this for one of my tests to make sure that the garbage is collected:
    use Test::More; # These tests try to make sure that objects are destroyed when they # fall out of scope; this requires avoiding circular strong references use strict; use warnings; #plan 'no_plan'; plan tests => 8; use Chemistry::File::Dumper; my $dead_atoms = 0; my $dead_bonds = 0; my $dead_mols = 0; { my $mol = Chemistry::Mol->read("t/mol.pl"); isa_ok( $mol, 'Chemistry::Mol' ); is( scalar $mol->atoms, 8, 'atoms before'); # make sure cloned molecules are also gc'ed my $mol2 = $mol->clone; # atom deletion garbage collection test $mol->atoms(2)->delete; is( $dead_atoms, 1, "delete one atom - atoms" ); is( $dead_bonds, 4, "delete one atom - bonds" ); is( $dead_mols, 0, "delete one atom - mols" ); } is( $dead_atoms, 16, "out of scope - atoms" ); is( $dead_bonds, 14, "out of scope - bonds" ); is( $dead_mols, 2, "out of scope - mols" ); sub Chemistry::Mol::DESTROY { $dead_mols++ } sub Chemistry::Atom::DESTROY { $dead_atoms++ } sub Chemistry::Bond::DESTROY { $dead_bonds++ }
Test::LectroTest and pseudo random distributions
by pernod (Chaplain) on Aug 04, 2005 at 14:40 UTC

    I'm off on a tangent here, so please excuse the off-topicness of this quick example.

    How do you test your shuffling or die rolling technique is fair and produces results according to your specifications (say, according to a Gaussing distribution), and isn't biased favouring certain outcomes?

    By using Test::LectroTest! Given this trivial implementation of an n-sided die, which returns a number between 1 and the number of sides on your die. Default number of sides is 6.

    package Die; sub new { my ( $pck, $sides ) = @_; return bless( { sides => $sides || 6 }, $pck ); } sub roll { my ( $self ) = @_; return int( rand( $self->{ sides } ) ) + 1; } 1;

    We then express how we want our die to perform with a Test::LectroTest property that generates a thousand tests with dies with from 1 to 100 sides. We roll the die, and check that the result is legal for this kind of die (eg. within the limits). The second propery generates a thousand six sided dies and rolls each of them once. The results are stored in the the LectroTest controller object with the tcon->label()-call, that automatically spits out the distribution of the roll.

    #! /usr/bin/perl use Test::LectroTest; use Die; Property { ##[ x <- Int( range => [ 1, 100 ], sized => 0 ) #]## my $die = Die->new( $x ); my $roll = $die->roll(); ( 0 < $roll && $x >= $roll ); }, name => "My die returns something between 1 and the number of sides + of the die."; Property { ##[ x <- Unit( 6 ) #]## my $die = Die->new( $x ); my $roll = $die->roll(); $tcon->label( $roll ); ( 0 < $roll && $x >= $roll ); }, name => "With a six sided die, I get this distribution";

    When I run this, I get:

    1..2 ok 1 - 'My die returns something between 1 and the number of sides of +the die.' (1000 attempts) ok 2 - 'With a six sided die, I get this distribution' (1000 attempts) # 18% 2 # 17% 3 # 16% 1 # 16% 5 # 16% 6 # 15% 4

    What this means is that the number 2 showed up 18% of the time, 3 17% etc. The numbers don't add up to a 100%, but I presume this is because of rounding in the presentation. In any case, this looks like acceptable behavior for a six sided die to me. I'm sure it's possible to automatically analyze the distribution and build this into the test, but I don't have the time to find out how right now. And please remember that this example was hacked together in a hurry, so I'm sure it can be improved.

    I like Test::LectroTest :)

    Edit: Removed an erroneous line in the second property.

    pernod
    --
    Mischief. Mayhem. Soap.

    Retitled by davido from 'OT: Test::LectroTest and pseudo random distributions'.

      No offence to Test::LectroTest intended, but in what way is this a test? For the "1 to 6" behavior, unless you're trying to test the underlying functions rand and int the only thing that matters is the endpoints of rand, which are 0 and a number close to 1 -- neither which Test::LectroTest are guaranteed to hit (and are even unlikely to hit). (See my module Test::MockRandom for a way to actually test your endpoints.) For the distribution, visually seeing it doesn't confirm anything, putting you right back to having to use one of the Statistics:: modules.

      I think I just don't "get" Test::LectroTest. To me, it feels like a thin veneer of testing -- "well, I tried it a bunch of times and it seemed to work". If the point is to identify edge/corner cases, it would seem to me to be better to identify and test them directly. If one isn't sure where these cases are, Devel::Cover will reveal them. (Note -- coverage is not correctness, but coverage will point out branches/conditions not taken in the code, which are the edge cases.)

      On reflection, maybe the point of Test::LectroTest is to try to expose the edge cases in your dependencies outside your own conditionals -- sqrt and division by zero come to mind. But I'd call it "stress testing" in that case and suggest that it is different from the way the term "testing" is usually meant in the various perl test suites. It doesn't tell you that your code is correct, only that it hasn't been shown to be incorrect for some number of trials.

      Test::LectroTest's author's presentation has a fairly good example using Email::Address, but includes a rather length set of email address generators (p. 49) which opens the question of how you know whether the error is in the generators or in the code?

      If someone's used Test::LectroTest extensively, I'm very curious to know in what kinds of cases it's proved useful.

      -xdg

      Code written by xdg and posted on PerlMonks is public domain. It is provided as is with no warranties, express or implied, of any kind. Posted code may not have been tested. Use of posted code is at your own risk.

        I think your critique of my simplistic example is valid, and suggest looking at the tutorial mentioned at the end of this node for a better demonstration of specification based testing. My point was using tcon->label() in a hackish way to find a distribution. Please file my ramblings under TIMTOWTDI :)

        On reflection, maybe the point of Test::LectroTest is to try to expose the edge cases in your dependencies outside your own conditionals -- sqrt and division by zero come to mind. But I'd call it "stress testing" in that case and suggest that it is different from the way the term "testing" is usually meant in the various perl test suites. It doesn't tell you that your code is correct, only that it hasn't been shown to be incorrect for some number of trials.

        I think that observation is correct. Specification based testing, which Test::LectroTest implements a framework for is based on the idea that you formulate the constraints. Then you leave it to the computer to try to violate your assumptions within the given constraints. This is one testing tool among many, and I find the technique useful, if only to allow myself to be humbled by my machine from time to time.

        If you haven't seen tmoertel's (Test::LectroTest's author) excellent tutorial, I suggest taking a look. It demonstrates why manually testing edge cases in some cases is not enough.

        Thank you for your comments!

        pernod
        --
        Mischief. Mayhem. Soap.

        xdg wrote
        I think I just don't "get" Test::LectroTest.
        Maybe this will help.

        I wrote LectroTest because I wanted an alternative to traditional, case-based unit testing that offered markedly different cost, benefit, and mental models for testing. There are many times when case-based testing sucks, and for these times LectroTest offers programmers another option. The two approaches complement each other and can even be seen as duals.

        The LectroTest approach requires programmers to be explicit about what their programs are supposed to do. Programmers must write property specifications that define the required behaviors of their programs. Then LectroTest uses random sampling to automate the process of gathering evidence to support (or refute) the claims made by the property specifications.

        Case-based testing, on the other hand, requires programmers to write individual test cases that each provides incremental evidence for (or against) some implied claim of overall correctness. Together, the test cases represent an implicit definition of correctness, but such definitions are usually difficult to extrapolate from the cases and often are nebulous and incomplete, which isn't necessarily a bad thing: in real life, formal notions of correctness may be hard to define.

        A table makes the salient differences easy to see:

        LectroTest Case-based testing
        Definition of correctness Explicit
        (via hand-written properties)
        Implicit
        (via manual extrapolation)
        Test cases Implicit
        (automatically generated)
        Explicit
        (written by hand)

        Which approach is best depends on what you are doing. As a rule of thumb, if you can easily specify what a piece of code ought to do, there's a good chance that LectroTest will be a great way to test that code. If, however, you are working on code for which correctness is a difficult concept to formalize, case-based testing will probably be the more fruitful approach.

        Cheers,
        Tom

      I like Test::LectroTest :)

      Me too.

      I think the difference between writing manual tests and using a testcase generator is somewhat analogous to using the CGI html generator functions and print, compared to using a templating module. It's another level of abstraction that ensures the details are accurate taken care of (given a good module), allowing greater productivity.

      It also avoids the tendancy to test the wrong things that is so prevalent.

      A relevant analogy is building cars. If the guys building engines stopped to physically verify that every nut they used was exactly the right size across the flats; made from steel containing the correct % of carbon; hardened to the approriate Stirling Rockwell number; and has threads who's profile was exactly as specified--car engines would cost the earth.

      Instead, the nuts are manufactured to a specified tolorance and sampled randomly for compliance against that specification.

      Way too many TDD programmers spend time testing the libraries they use (including Perl itself), with tests that they duplicate in every module/program they write, and that are also duplicated by many other programmers. These tests are (should be) done by the authors of those modules. If there is any doubt that the module/library/program is well-tested, then it can be verifed (once) by the organisation before certifying it for use, but there after, programmers within that organisation should rely upon it to comply with it's specification. Not re-test the same things over and over in every program that uses it.

      That's the essence of DBC.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
      "Science is about questioning the status quo. Questioning authority".
      The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://480791]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chilling in the Monastery: (9)
As of 2014-09-22 06:30 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (182 votes), past polls