Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re^3: Toggling test plans with vim

by Stoffe (Sexton)
on Aug 11, 2006 at 00:24 UTC ( #566728=note: print w/ replies, xml ) Need Help??


in reply to Re^2: Toggling test plans with vim
in thread Toggling test plans with vim

Yes good point, as it was formulated. That was clumpsy.

But why is there a need to know the exact amount of tests beforehand? Someone mentioned catching dieing or exits, but that should be easy enough to detect. I suppose it's all down to what philosophy lies behind the test systems from the start: most other systems I've seen have setup and teardown, so it is easy to tell if eveything ran (died after test # 12). Note that this does not mean you have to have a teardown, that's in the base class. Also, they seem to catch any exits and other stuff, so you can get a pass, a fail or an error out of a test, and they usually keep on running the rest. Also quite easy to track.

Is it not possible to have a runner of that kind in Perl, or is it just that everyone sticks with the old stuff? It seems very un-lazy to have to run around updating that number all the time, counting properties manually and so on. I do not see the win.


Comment on Re^3: Toggling test plans with vim
Re^4: Toggling test plans with vim
by adrianh (Chancellor) on Aug 12, 2006 at 23:10 UTC
    But why is there a need to know the exact amount of tests beforehand? Someone mentioned catching dieing or exits, but that should be easy enough to detect.

    Not so easy :-)

    Normal exceptions, exiting with a non-zero value, etc. are detected. Unexpected crashes from XS code, POSIX::_exit, or exit(0) are not. See below for why.

    . I suppose it's all down to what philosophy lies behind the test systems from the start: most other systems I've seen have setup and teardown, so it is easy to tell if eveything ran (died after test # 12).

    It's not really that aspect that causes the problem. Indeed Perl has a couple of testing frameworks that use the xUnit model (Test::Unit and my Test::Class).

    The issue is that in frameworks like JUnit, TestNG, etc. the test runner and the tests are running in the same process. If the tests exit for an unexpected reason then nothing runs - so hard crashes of the system are trivial to detect (hey - my test suite has crashed!)

    In Perl the test runner (Test::Harness) and the tests (the *.t files) run in separate processes.

    This is good in many ways, because it makes our tests more flexible (e.g. we can run tests in different programming languages) and has the advantage that a hard crash will not stop all the other tests running.

    The disadvantage is that the test runner has to figure out for itself whether that test script exited because all the tests had run, or whether it exited because something unexpected happened.

    One way it does this is via the fixed plan. Double checking that the number of tests that were expected to run actually ran.

    It seems very un-lazy to have to run around updating that number all the time, counting properties manually and so on. I do not see the win.

    The win is that we have a more flexible testing framework by splitting the test runner and the test producer up.

    The cost is that detecting unexpected failures is harder. There's been some conversation on the perl-qa list recently on just this topic, with some ideas that might make the 'no_plan' option even safer.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://566728]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others having an uproarious good time at the Monastery: (7)
As of 2014-10-31 05:38 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    For retirement, I am banking on:










    Results (214 votes), past polls