Beefy Boxes and Bandwidth Generously Provided by pair Networks Frank
Syntactic Confectionery Delight
 
PerlMonks  

Re: Failing tests of Test::TCP on Windows

by tobyink (Abbot)
on Mar 05, 2013 at 09:52 UTC ( #1021785=note: print w/ replies, xml ) Need Help??


in reply to Failing tests of Test::TCP on Windows

I can't answer the Test::TCP-related questions, but with regard to "is it enough to run the test of Perl modules just once or how many times are they needed to be run?" I can venture an opinion.

The fact that a test passes once, does not prove it will always pass. Take a look at this...

$ date Tue Mar 5 09:30:54 GMT 2013 $ perl ~/tmp/scratch.pl 1..1 ok 1 - DateTime returns a value for "second" method $ perl ~/tmp/scratch.pl 1..1 ok 1 - DateTime returns a value for "second" method $ perl ~/tmp/scratch.pl 1..1 not ok 1 - DateTime returns a value for "second" method # Failed test 'DateTime returns a value for "second" method' # at /home/tai/tmp/scratch.pl line 7. # Looks like you failed 1 test of 1.

What was the test case? Spot the obvious error...

use strict; use warnings; use Test::More tests => 1; use DateTime; ok( DateTime->now->second, 'DateTime returns a value for "second" method', );

That's right, the test doesn't check the definedness of DateTime->now->second; it tests its truthfulness. When the clock hits an even minute, the second method correctly returns 0.

However, datetime-specific issues and race conditions can be very unpredictable. There are also all kinds of subtle bugs that can crop up running test cases in parallel with each other or in a random order.

These kind of errors are quite rare, and you may need to run the test suite hundreds of times to catch them. So in most cases the cost-benefit ratio doesn't justify running the test suite multiple times as part of the installation process.

Usually better would be a continuous integration policy, where the test suite is run before each code checkin, with test case regressions noted in the commit message. (And ideally should pass the suite before checkins are merged into the master/default/trunk codebase.) A policy like this should catch a large number of these sporadic errors.

package Cow { use Moo; has name => (is => 'lazy', default => sub { 'Mooington' }) } say Cow->new->name


Comment on Re: Failing tests of Test::TCP on Windows
Select or Download Code

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1021785]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others browsing the Monastery: (11)
As of 2014-04-17 18:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    April first is:







    Results (453 votes), past polls