Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine

Re: Failing tests of Test::TCP on Windows

by tobyink (Abbot)
on Mar 05, 2013 at 09:52 UTC ( #1021785=note: print w/replies, xml ) Need Help??

in reply to Failing tests of Test::TCP on Windows

I can't answer the Test::TCP-related questions, but with regard to "is it enough to run the test of Perl modules just once or how many times are they needed to be run?" I can venture an opinion.

The fact that a test passes once, does not prove it will always pass. Take a look at this...

$ date Tue Mar 5 09:30:54 GMT 2013 $ perl ~/tmp/ 1..1 ok 1 - DateTime returns a value for "second" method $ perl ~/tmp/ 1..1 ok 1 - DateTime returns a value for "second" method $ perl ~/tmp/ 1..1 not ok 1 - DateTime returns a value for "second" method # Failed test 'DateTime returns a value for "second" method' # at /home/tai/tmp/ line 7. # Looks like you failed 1 test of 1.

What was the test case? Spot the obvious error...

use strict; use warnings; use Test::More tests => 1; use DateTime; ok( DateTime->now->second, 'DateTime returns a value for "second" method', );

That's right, the test doesn't check the definedness of DateTime->now->second; it tests its truthfulness. When the clock hits an even minute, the second method correctly returns 0.

However, datetime-specific issues and race conditions can be very unpredictable. There are also all kinds of subtle bugs that can crop up running test cases in parallel with each other or in a random order.

These kind of errors are quite rare, and you may need to run the test suite hundreds of times to catch them. So in most cases the cost-benefit ratio doesn't justify running the test suite multiple times as part of the installation process.

Usually better would be a continuous integration policy, where the test suite is run before each code checkin, with test case regressions noted in the commit message. (And ideally should pass the suite before checkins are merged into the master/default/trunk codebase.) A policy like this should catch a large number of these sporadic errors.

package Cow { use Moo; has name => (is => 'lazy', default => sub { 'Mooington' }) } say Cow->new->name

Log In?

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://1021785]
and all is quiet...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2018-05-27 02:01 GMT
Find Nodes?
    Voting Booth?