RFC: Tutorial on Testing

[tmoertel]

I recently wrote an automatic, specification-based testing system for Perl called LectroTest. Because specification-based testing takes a while to grok, I'm trying to write some helpful introductory documentation. One bit o' docs I just finished is a tutorial-style introduction to testing that works through Test::More and ends up at LectroTest.

But I have a problem. I wrote the tutorial and, worse, I wrote LectroTest. I know how the darn thing works. If my tutorial has holes in it (or gaping, thorn-filled chasms for that matter), I won't notice because I'll subconsciously fill in the blanks.

That's where I need your help. Would you mind taking a critical read of my tutorial and giving me some feedback? It might even be fun. (If you're into that kind of thing – or if you just want to mock my writing.)

To all those who dare to venture within, many thanks!

In this tutorial we'll take a look at testing our Perl code. First, we'll quickly review why it's a good idea to test. Then, we'll create a small program to test and test it with Test::More. Next, we'll test our program more extensively with the relatively new LectroTest system. Finally, after a quick review, we'll grab an espresso and return to coding with renewed vigor.

Why test?

Let's say we are writing a program. Being hackers of good character, we naturally want our program to be correct. In other words, we want to be confident that our program will behave as we expect it to behave. One way of gaining that confidence is through testing.

Testing is a practice in which we compare our program's behavior to its expected behavior on a case-by-case basis. We, the humans who know what our program is expected to do, create the test cases. Each test case says, "For this particular set of conditions, the expected behavior of our program is that." We then run our program under the given set of conditions, observe its behavior, and compare what we observed to the expected behavior for the test case. If the observed and expected behaviors match, the program is said to have passed the test. Otherwise, it fails, and we ought to examine our code for errors.

It's easy to test in Perl with Test::More

It's easy to create test cases in Perl. To see how the process works, let's first write some code to test. Below is an implementation of angdiff, a subroutine that computes the difference between two input angles, a and b, given in degrees. The expected result is the smallest positive angle between the two inputs. For convenience, we'll put angdiff in its own module called AngularDifference:

# File: AngularDifference.pm

package AngularDifference;

BEGIN {
    use Exporter;
    our @ISA = qw( Exporter );
    our @EXPORT = qw( &angdiff );
}

# compute the difference between angles $a and $b

sub angdiff($$) {
    my ($a, $b) = @_;
    return abs($a - $b) % 360;
}

1;

Take a look at the code for angdiff. It takes the absolute value of the difference between $a and $b and then "clamps" the result modulo 360 because two angles can never be more than a full circle -- 360 degrees -- apart.

It seems straightforward enough, but how confident are we that there's not a subtle error lurking in the code? Let's create some test cases to raise our confidence.

The Test::More module was created to make this kind of testing easy:

# File: AngularDifference.t

use AngularDifference;  # provides angdiff
use Test::More tests => 6;

# identical angles should have a diff of zero
is( angdiff(  0,  0),   # observed result
    0,                  # expected result
    "zero at 0"         # name of the test case
);                                           # case 1
is( angdiff( 90, 90),   0, "zero at 90" );   # case 2

# order of angles shouldn't matter to diff
is( angdiff(  0, 45),  45, "0,45 -> 45" );   # case 3
is( angdiff( 45,  0),  45, "45,0 -> 45" );   # case 4

# should return the smallest angle between
is( angdiff(  0,270),  90, "0,270 -> 90, not 270" );  # case 5

# multiples of 360-degrees shouldn't matter
my ($a,$b) = (360 * 2, 360 * 4);
is( angdiff($a,$b+23),23, "$a,$b+23 -> 23" );  # case 6

Here we have created a suite of six test cases. The suite, itself, is nothing more than a small Perl program that uses Test::More.

First, we imported the code we wanted to test, which is in the AngularDifference package. Then we loaded Test::More and told it that we have six test cases.

Next comes the test cases. Each takes the form of an is statement that compares the observed output of angdiff with the expected, correct result for the given set of conditions. The first test case is commented to show each part clearly. Each case also has a name. Always provide good names for your test cases because it makes your life easier in the long run, especially when your test suites become large.

Note that our small suite includes cases designed to flush out common programming errors that might be lurking around zeroes and argument ordering. We also have a few domain-specific cases that deal with properties of angles and circles. The idea is to exercise our code and make it sweat.

To run the tests, just run the program:

$ perl AngularDifference.t
1..6
ok 1 - zero at zero
ok 2 - zero at 90
ok 3 - 0,45 -> 45
ok 4 - 45,0 -> 45
not ok 5 - 0,270 -> 90, not 270
#     Failed test (AngularDifference.t at line 15)
#          got: '270'
#     expected: '90'
ok 6 - 720,1440+23 -> 23
# Looks like you failed 1 tests of 6.

Looks like we have a problem! Our implementation of angdiff failed the 5th test case. We asked it to compute the difference between 0 and 270 degrees, and it returned 270. However, the correct result is the smallest angle between 0 and 270, which is 90 degrees.

It looks like our intuition to use modulo-360 truncation was wrong. Now that we think about it, we can't ever have a difference in angles greater than 180 degrees because, if we did, we could always find a shorter difference by going the other way around the circle from a to b. So let's bring the truncation threshold down to 180 instead:

sub angdiff($$) {
    my ($a, $b) = @_;
    return abs($a - $b) % 180;
}

And now, let's re-run our tests:

$ perl AngularDifference.t
1..6
ok 1 - zero at zero
ok 2 - zero at 90
ok 3 - 0,45 -> 45
ok 4 - 45,0 -> 45
ok 5 - 0,270 -> 90, not 270
ok 6 - 720,1440+23 -> 23

Ah, that looks better. Our revised implementation passes all six test cases.

Automatic, specification-based testing with LectroTest

With a small investment of six test cases, we were able to find a problem in our implementation of angdiff. What if we considered more test cases? What if we considered all possible test cases? If we could show that the actual and expected behaviors of our implementation were identical for all cases, we would actually prove that our implementation was correct. We would earn the ultimate kind of confidence!

Unfortunately, using testing to prove a program's correctness is impractical for almost all real-world programs. The typical program's behavior represents a surprisingly vast space to test, and there is no reasonable way to test all of the possible cases within it.

But we can test samples from that vast space to gain some degree of confidence in our program's correctness. Many common programming mistakes pollute a large portion of the "test space" with detectable deviations from expected behavior. The more samples we take, the more likely we are to find these deviations and the more reason we have to be confident in our program when we don't find them.

Right now, we take six samples from the test space of angdiff's behavior. That's not many and probably shouldn't give us a strong sense of confidence in our implementation. If we want more confidence, we will need more cases.

How many is enough? That's hard to say. Maybe we could analyze our code, figure out exactly how it works, and create just the right amount of just the right kinds of cases to give the code a good workout. But that would be tricky, especially if our code were more complicated. And what if we made a mistake and missed some important cases?

Let's consider another approach. Instead of creating just enough of just the right kind of cases, what if we created hundreds of mediocre cases? If we took that approach, we would have so many cases that each wouldn't need to be "just right." But where would we get all of those tests?

It sure would be nice if we could "delegate" this job to our computers. Hmmm...

Let's think about that idea. Consider our second case from earlier:

is( angdiff( 90, 90),   0, "zero at 90" );   # case 2

We can interpret it as saying, "For $a=90 and $b=90, we assert that angdiff($a,$b) must be 0." Wouldn't it be great if we could generalize that claim? We would like to be able to say, "For all $a and $b, we assert that angdiff($a,$b) must be X."

But there's a rub. It's that pesky placeholder for the correct result, X. If we were given some random angles $a and $b to check as a test case, we could certainly determine the actual result of angdiff($a,$b) – just call the function. But how could we determine the expected, correct result? That's tricky. If we could do that, we wouldn't need to test angdiff in the first place: Our code for determining the correct result would be the correct implementation of angdiff!

But not all is lost. What if we could change the way we look at the "test space" so that we could construct any random test case and the correct result for that case at the same time?

If we picked any angles $a and $b, we would have the problem of determining the correct difference between them – back to square one. But, if we instead picked $a and the correct, expected difference first, we could then determine a corresponding $b by working backward. Then we would know $a, $b, and the correct difference. That's everything we need for a test case!

Let's formalize this plan as a recipe:

1. Pick a random angle $a.
2. Pick a random difference $diff, in the range -180 to 180.
3. Compute $b = $a + $diff.
4. Now we have a test case: We assert that angdiff($a,$b) must equal abs($diff).

If you think about it, our recipe above is actually a specification of a general property that our implementation must hold to: "For all angles a and for all angles diff in the range -180 to 180, we assert that angdiff($a, $a + $diff) must equal abs($diff)."

This is where LectroTest comes in. It is an automatic, specification-based testing system. It is designed to take property specifications (like the one we just created) and check them by running large numbers of random trials against the software we're testing. Each trial is an attempt to "break" one of our property assertions at a particular point in the test space. Because LectroTest is automated, it can quickly and painlessly check thousands of test cases for us, giving us a much higher degree of confidence than is practical with manual test cases.

To see how LectroTest works, let's convert our recipe into a real, live LectroTest property specification that we can check. Like before with Test::More, we create a simple Perl program to hold our properties. This time, however, we use Test::LectroTest and declare a property instead of individual test cases:

# File: AngularDifference.l.t

use AngularDifference;
use Test::LectroTest;

Property {
    ##[ a <- Int, diff <- Int(range=>[-180,180]) ]##
    angdiff($a, $a + $diff) == abs($diff)
}, name => "angdiff holds to defn of angular difference";

The first part of the property specification is a generator binding. It tells LectroTest to automatically set up variables for you to use in a behavior test that comes later:

    ##[ a <- Int, diff <- Int(range=>[-180,180]) ]##

It reads, "For all integers a and for all integers diff in the range -180 to 180." Sound familiar? The only twist is that we are representing angles as integers for convenience.

The second part of the property specification is a behavior test. It uses the variables we bound earlier to test whether angdiff has the expected behavior at a particular instance of a and diff in the test space:

    angdiff($a, $a + $diff) == abs($diff)

Note that the behavior test is just a block of code that has a true or false result. True means that angdiff had the expected behavior and thus passed the test for this particular case of a and diff. False means that it failed the test.

Finally, like before, we provide a meaningful name. Because our property is general and derived from the mathematical definition of angular difference, we name it "angdiff holds to defn of angular difference".

Now let's run our property check. Again, we just run the program:

$ perl AngularDifference.l.t
1..1
not ok 1 - 'angdiff holds to defn of angular difference' falsified in 
+535 attempts
# Counterexample:
# $a = 148;
# $diff = 180;

Oops! LectroTest was able to falsify our property claim. That means it was able to find a point in the test space where our claim didn't hold for our revised angdiff implementation. It also emitted a counterexample, which shows exactly where that point is.

We can plug the counterexample into our code to debug the problem. After we fix the problem, we can add the counterexample to a list of test cases for regression testing to make sure that future modifications to angdiff don't reintroduce the same erroneous behavior.

By examining the counterexample, we see that angdiff "broke" when $diff was set at 180 degrees. Looking back at our new angdiff code, we can see the problem: Our modulo-180 truncation wraps sharply when the difference increases to 180 degrees, when it shouldn't. Let's compute a small table by hand that shows the correct differences for various values of $b when $a is fixed at 0 degrees:

When $a = 0

And $b is    The expected   And angdiff
             result is      returns
=========    ============   ===========
  150           150            150
  160           160            160
  170           170            170
  180           180              0  <-- Oops
  190           170             10  <-- Oops
  200           160             20  <-- Oops
  210           150             30  <-- Oops

See how the expected result climbs up toward 180 and then starts down again? See how angdiff wraps around sharply? That's the problem. With this knowledge, we can fix the bug:

sub angdiff($$) {
    my ($a, $b) = @_;
    my $delta = ($a - $b) % 360;
    return $delta > 180 ? 360 - $delta : $delta;
}

Let's repeat our property check by re-running the program:

$ perl AngularDifference.l.t
1..1
ok 1 - 'angdiff holds to defn of angular difference' (1000 attempts)

Ah, now that is more like it!

Still, having been burned once before by overconfidence in our testing, we should be cautious. Yes, LectroTest was able to find a problem that our manual test cases didn't, but do we have reason to believe there aren't more errors in hiding?

Maybe we should try to quantify the kinds of test cases that LectroTest is creating for us behind the scenes. In our property specification, we can make use of the magic object $tcon, which LectroTest provides to let us interact with the test controller. One of the things we can ask the test controller to do is attach labels to individual trials. At the end of our property check, LectroTest will tabulate the trials based on the labels we have attached and provide us with summary statistics.

One thing we might want to examine is how far apart the input angles $a and $b are. (Remember, we're letting $b = $a + $diff, so $diff tells us how far apart $a and $b are.) Here's one way to categorize them:

Property {

    ##[ a <- Int, diff <- Int(range=>[-180,180]) ]##
 
    if    ($diff == 0)         { $tcon->label("zero") }
    elsif (abs($diff) <= 90)   { $tcon->label("1 to 90") }
    elsif (abs($diff) <= 180)  { $tcon->label("91 to 180") }
    elsif (abs($diff) <= 360)  { $tcon->label("181 to 360") }
    else                       { $tcon->label("> 360") }

    angdiff($a, $a + $diff) == abs($diff)

}, name => "angdiff holds to defn of angular difference";

Re-running our check reveals the statistics:

1..1
ok 1 - 'angdiff holds to defn of angular difference' (1000 attempts)
#  58% 1 to 90
#  41% 91 to 180
#   0% zero

Looking at the statistics tells us that we aren't checking any cases were the input angles are farther than 180 degrees apart. This seems like a hole in our testing strategy because in the real world, angles can be farther apart than that. The problem is that we constrain our $diff to a 180-degree magnitude, so that's as far apart our input angles will ever be.

To introduce a greater spread, we can add random multiples of 360 degrees to our calculation for the second input angle $b. Such multiples won't affect our assertion that the expected result is abs($diff), which is what makes our testing strategy work. Here's the rewritten property:

Property {

    ##[ a <- Int, n <- Int, diff <- Int(range=>[-180,180]) ]##
 
    my $b = $a + 360*$n + $diff;
    my $delta = abs($b - $a);

    if    ($delta == 0)    { $tcon->label("zero") }
    elsif ($delta <= 90)   { $tcon->label("1 to 90") }
    elsif ($delta <= 180)  { $tcon->label("91 to 180") }
    elsif ($delta <= 360)  { $tcon->label("181 to 360") }
    else                   { $tcon->label("> 360") }

    angdiff($a, $b) == abs($diff)

}, name => "angdiff holds to defn of angular difference";

Now, let's run our check yet again and examine the frequencies:

1..1
ok 1 - 'angdiff holds to defn of angular difference' (1000 attempts)
#  98% > 360
#   0% 181 to 360
#   0% 1 to 90

Now it seems that we have the opposite problem. We're testing the really large differences most of the time, but not the smaller ones. Nevertheless, we are testing the lower ranges, if just a little, because even though they have a 0% frequency, the fact that they show up in the list at all means they were tested at least once.

Still, let's do the right thing and try to re-balance the distribution evenly. What unbalanced it was the introduction of $n, which is multiplied by 360. Any time $n is greater than one, we'll be in the "> 360" case. Why not, then, make $n pick a small integer half of the time and a large integer the other half? That way, we'll get an even distribution among our categories.

Believe it or not, this kind of thing is easy to do in LectroTest. We can use a special generator combinator OneOf to combine simple generators into a more complex one that does what we want:

n <- OneOf( Int(range=>[-1,1]), Int )

As you might expect from its name, OneOf chooses one of the generators we've given it at random, and uses that generator to generate its final result. So we're choosing between a small-range generator and an unconstrained generator. Putting it all together, we get the following, revised property:

Property {

    ##[ a    <- Int,
        n    <- OneOf( Int(range=>[-1,1]), Int ),
        diff <- Int( range=>[-180,180] )  ]##
 
    my $b = $a + 360*$n + $diff;
    my $delta = abs($b - $a);

    if    ($delta == 0)    { $tcon->label("zero") }
    elsif ($delta <= 90)   { $tcon->label("1 to 90") }
    elsif ($delta <= 180)  { $tcon->label("91 to 180") }
    elsif ($delta <= 360)  { $tcon->label("181 to 360") }
    else                   { $tcon->label("> 360") }

    angdiff($a, $b) == abs($diff)

}, name => "angdiff holds to defn of angular difference";

Running the property check:

1..1
ok 1 - 'angdiff holds to defn of angular difference' (1000 attempts)
#  67% > 360
#  17% 181 to 360
#   9% 1 to 90
#   6% 91 to 180
#   0% zero

That's better, but we're still placing too much emphasis on the large differences. Well, we can take care of that, too. Let's replace the OneOf combinator with Frequency, which lets us hand-tune the frequencies with which its sub-generators are chosen. We'll give the small-range generator a 20-to-1 advantage:

n <- Frequency( [20,Int(range=>[-1,1])], [1,Int] )

With this change in place, we get more-agreeable coverage:

1..1
ok 1 - 'angdiff holds to defn of angular difference' (1000 attempts)
#  37% > 360
#  32% 181 to 360
#  18% 1 to 90
#  10% 91 to 180
#   0% zero

With this result, we have good reason to be confident that our implementation of angdiff is correct. We created a property that specified the expected behavior of our implementation for all possible inputs. We tested the implementation against our specification by running thousands of test cases that were distributed randomly throughout the overall test space. Further, we quantified the distribution of cases that we were testing to ensure that there were no holes.

Mission accomplished! (Now would be a good time to enjoy a celebratory espresso.)

Let's review

Testing is an effective technique for improving our confidence in the software we write. Lucky for us, Perl makes it easy to create good test suites.

Test::More gives us a simple, low-overhead way to test how our software behaves in specific cases that we define by hand. In many cases, this is all we will need.

When we do need more, one option is use LectroTest's specification-based tests. LectroTest lets us to specify behaviors that ought to hold across large test spaces, and then it randomly samples those spaces to see whether the properties actually hold. It can't prove that our properties hold, but for many kinds of properties it can quickly and easily give us good reason to be confident in them.

To ensure good coverage when we use a tool like LectroTest, it's a good idea to understand how the tool is sampling the test spaces we give it. Labeling is an easy way to quantify the sampling distribution. If it turns out that we need to adjust the distribution, generator combinators like OneOf and Frequency provide the tuning knobs we need.

Sources for more information

If you're interested in checking out any of these tools – and you ought to because we only scratched the surface here – they're easy to find. Test::More is probably included in your Perl installation already, out of the box. LectroTest is available as Test::LectroTest on CPAN, and you can find more information about it at the LectroTest Home. Both tools provide much more depth than we covered here.

20040914 Edit by tmoertel: Fixed typos and some usage problems. Thanks, schodckwm!

20040914 Edit by castaway: Changed title from 'Care you lend me your widsom? And your eyes, too?'

[stvn]

To start out, let me say that I was very excitied to see Test::LectroTest come out on CPAN, I have recently stumbled onto specification based testing myself (see here for my rant on the subject). So I am looking forward to experimenting with your module myself.

One thing I would like to see in your tutorial, is more information about how to test with input other than integers. In my node on this subject, I showed a specification based test for a list, how would I do something like that with Test::LectroTest? Can I feed it a sampling of string data, which can be randomized/shuffled in some way? Nested lists? (I see in Test::LectroTest::Generator where I would find some of this info, but it would still be nice in tutorial form)

I would also like to see how to make more complex data-structures (structure being the key word there). For instance, I have a module Tree::Binary which I actually tried to do some specification based testing with (see the file 50_Tree_Binary_Theorem_Proofs_test.t for details). How would I go about using Test::LectroTest to create the random binary trees? Can I create a custom generator with Test::LectroTest? If so, how could I got about it?

I know some of these things I am talking about are more advanced uses of your module, but to be honest, I think your module really is going to appeal to a crowd which is likely to use those features. Specification based testing is not a beginners subject, and IMO has a somewhat steep learning curve (at least for us non-math/CS majors out here). I would suggest keeping that in mind and not be afraid to write the more advanced stuff.

[tmoertel]

I am looking forward to experimenting with your module myself.

Great! Please let me know if you have any suggestions or criticisms. (You might also want to check the TODO and see what's already in the pipeline.)

One thing I would like to see in your tutorial, is more information about how to test with input other than integers. In my node on this subject, I showed a specification based test for a list, how would I do something like that with Test::LectroTest? Can I feed it a sampling of string data, which can be randomized/shuffled in some way? Nested lists?

That's a good idea. You can do all of those things with LectroTest, but the tutorial doesn't even hint at it. In the Generator docs, as you mention, I give some examples of list-of-list and of tree building, and in my "Free Unit Tests In Perl with LectroTest" talk for the festive Pittsburgh Perl Mongers, I showed how to build RFC 2822–compliant email addresses. You're right, I ought to cover this in the tutorial. Even if for compactness reasons I can't add a full-blown example of these uses, at least I can mention them in passing and provide pointers to relevant portions of other docs.

Thanks again for your feedback.

[stvn]

Even if for compactness reasons I can't add a full-blown example of these uses, at least I can mention them in passing and provide pointers to relevant portions of other docs.

You might consider 2 tutorials (if you have time of course). A gentle introduction is nice, but a "jump right in the deep end" would also be nice too. I would even be happy to help out once I get my head around your module more.

Thanks again for your feedback.

No problem, thanks for the module, you saved me from writing it :)

[ww]

I'm not qualified to judge code, but from a wordmonger who's all too often frustrated with documentation, congratulations.

Both your conversational style and clear content made reading this a pleasure.

I have just minor tics/suggestions (possibly pure taste issues: ignore at will):

1. you might want to use elipsis with a period in the graf beginning:

   We can interpret it as saying,

   at:

   assert that angdiff($a,$b) must be ..."
   
   But there's a rub. It's that pesky "..." part.
   

   -- this is not a punctuation-nazi suggestion. The fourth dot (in each case) would make locating the reference easier (and might argue for dropping the space, word and period after the closequote.

2. This, however, may be useage-nazi... but, risking that, I suggest it would be better to say

   "so that's as far apart...."
   

   or

   "so that's the farthest apart...."
   

   in

   The problem is that we constrain our $diff to a 180-degree magnitude, so that's all the farther apart our input angles will ever be.

3. and, a minor typo:

   because even though they have a 0% frequency, the fact that the show u
   +p in the list at all means they were tested at least once.
   

   where "the" before "show up" is probably intended to be "they...."

again, ++!

[tmoertel]

That's exactly the kind of feedback I need. And, in fact, I have just applied all three of your suggestions to my draft. Thanks again for the sharp-eyed proofreading!

[pernod]

Great tutorial, and brilliant work! I completed your tutorial and read the rest of the documentation before I went beyond your celebrational espresso. The following is a condensed and simplified example of how I put your module to use in combination with chromatic's excellent Test::MockObject to test one of my own modules. Note that the enclosed snippet is untested, as i stripped away quite a bit of detail that I thought complicated the example.

In the web app I work on at $firm, we channel all database access through a single component. The module I want to test queries the database and ranks entities according to certain criteria, using a simple formula. What I want to test here is the method that does the ranking.

Read more... (2 kB)

When it comes to answering your question about oversights in the tutorial, then I agree with stvn that you should give the generators more focus. Especially some help on how to limit ranges, and combining generators into complex structures. I found the answers I needed in the the Test::LectroTest::Generator manual, though, so it worked great as a quick start.

I wondered where the name came from too, but after I found:

The result is LectroTest, a horribly named, automatic, specification-based testing tool for Perl.

on your homepage, I thought it better not to ask ;)

All in all, I believe Test::LectroTest will become a valuable tool at work. Thank you very much :)

[tmoertel]

Thanks for your feedback on LectroTest, and your example with Test::MockObject is a cool use. (BTW, I don't think that calculating the expected score in the Property is an abuse at all.)

If you don't mind a suggestion, you could simplify the data-building portion of your code by letting LectroTest build the more-complicated data structure for you:

#!/usr/bin/perl

use strict;

use Data::Dumper;
use Test::LectroTest trials => 10;

use List::Util qw( reduce );

$Data::Dumper::Terse = 1;

Property {

    ##[ data <- List( Hash( Unit(1)
                          , Int( range=>[50,100], sized=>0 )
                          , length=>1 )
                    , length=>5 )
    ]##

    my @input = map { values %$_ } @$data;
    my $questions = reduce { $a + $b } @input;

    print STDERR Dumper( { data      => $data
                         , input     => \@input
                         , questions => $questions } ), "\n";

    1;  # always passes trials

}, name => "shell property that holds a generator example";

Also, for the Int generator, you shouldn't specify a range that does not contain zero if the generator is sized. The reason for this rule is that at run time, the intersection of the sizing-guidance range and your given range can be empty, thus making it impossible to generate a valid value. (I have updated the LectroTest::Generator docs to reflect this, and LectroTest will now complain if you try to combine these settings.)

Thanks again for taking the time to provide feedback!

Cheers,
Tom

[pernod]

Thank you! I will incorporate your suggestions, as they are both more concise and more elegant, not to say easier to read. Perhaps a cookbook section with examples on different standard structures might be an idea?

Anyhow, this is great input for future fiddling with the settings. I'll study the docs for sizing-guidance, as I haven't quite figured those out yet.

[BrowserUk]

I skipped by this the first time around. I think your original title put me off. I'm very glad it was re-titled. I'm even more glad that I took the time to read it.

This is a test tool worthy of occupying space in the rather lack lustre Test::* namespace--even if the name itself is somewhat mysterious :)

I think that this could easily have been called Test::Smart. In keeping with a local advertising slogan that reads: "Work smarter, not harder".

It's too early days yet in my understanding of Test::LectroTest (So good, it tests twice many times? :), for me to have found the answers to these questions myself, so I'll ask them here before going on to read more:

1. It feels like it would be possible to combine the logic of Lectro into the actual code such that the generator bindings could double serve as parameter verifications?

   My thoughts here are that if the bindings were embedded within the functions themselves and served as parameter checks (possibly disable-able for production), they are more likely to stay in step with changes in the specification of the function over time.

2. I've had a fairly long history with using testcase generators (some further infoRe: Software Design Resources & Re: Re: Re: Software Design Resources etc.), and with writing them. One of the major benefits of using (directed) random testcase generators is that it is possible to infer some "measure of goodness" statistically from the number of testcases run -v- bugs found.

   For this to be properly effective, it requires not just the overall tests run count, but an analysis of those tests to produce a measure of coverage. For a function taking integers as input, the range is finite and quantifiable, and by accumulating the actual values used in generated tests, it becomess possible to derive a coverage statistic. This is much harder for other types of parameter with continuous input ranges, but even these can often be quantified on some basis, relative to the code under test.

   The key here is that it requires that generated parameters be logged and acculmulated. Is there any intent to provide this type of facility?

3. Finally, your chosen example is very good. It allows you to demonstrate the benefits of the approach with something that is apparently simple, but for those of use that have forgotten the schoolboy diagrams we drew that showed just how inconvenient it is doing moduler math on angles, it allowed us to be delighted by the step-by-step revelations :)

   However, in it's simplicity, it suffers from not showing how difficult it can be to choose the correct ranges for sampling. I'm going on to read the rest of the docs, but do you have any advise/tutorials on how to go about selecting ranges for where the inputs are more complex (strings, arrays, hashes etc.)?

Overall, The tutorial was extremely readable--which I consider very important in such material; methodical accuracy is of little good if noone reads it--, and a damn good advert for the both the module, but more importantly, the methodologyy it uses.

I've expressed my doubts over the efficacy of the types of testing possible using most of the Test::* modules. Part of the problem is that this is another of those areas where "more" rarley equates to "better".

It is also the case that when tests are coded by the same person that writes the code, they tend to concetrate the tests on those areas of the code that they spent most time thinking about. Often this works out to be the areas that they had most trouble coding. Inevitably, it's the areas that they spent least time thinking about that need most testing.

In an ideal world, we would all have a test engineer to design/code our tests for us, but that is an increasingly rare situation. Using a test module that takes input in a generalised form and then randomly distributes the tests is a real step forward (IMO).

If this methodology can be combined with DBC constraints, it would further lift the testing away from the assumptions of the coder. Allowing the designer (often as not, the coder wearing a different hat) to specify the function in terms of constraints applied to the inputs, and then allow a module such as this to take over the testing from there.

Ultimately, if the actual test values could be accumulated and analysed, you would have the basis of a viable quality metric.

[chanio]

GREAT!

Your way of exposing caught me reading until the last comment!

My guess is that this work is not only limited to mathematics or stadistics use.

I can't anyway, yet understand why diff has to be abs :) in your test.

I also find it difficult to understand the way of using

OneOf( Int...

and then changing it for

Frequency( [20,Int(range=>[-1,1])]...

And where do those 20 to 1 successfully come from :) ...

It would help to have some 'lateral thinking' equivalences always intermingled in the tutorial. Like using words as 'paradigm'. Or using more graphical examples when showing the quantification of the kinds of test cases

I know that using maths is a way of understanding the world. And that it allows mathematicians to imagine how it might apply to specific cases.

But I think that it is a pity that it would only be limited to those people.

In its perl nature, it doesn't need to look as a complex thing.

[water]

Wow. This is really cool. tmoertel++. Powerful test module, great docs. Given your writing style, I'd wager you have a math degree in your background. The formalism and clarity is much appreciated.

It is this sort of contribution that makes CPAN and Perl so unique and powerful. Thanks!