Re^4: Toggling test plans with vim

[adrianh]

But why is there a need to know the exact amount of tests beforehand? Someone mentioned catching dieing or exits, but that should be easy enough to detect.

Not so easy :-)

Normal exceptions, exiting with a non-zero value, etc. are detected. Unexpected crashes from XS code, POSIX::_exit, or exit(0) are not. See below for why.

. I suppose it's all down to what philosophy lies behind the test systems from the start: most other systems I've seen have setup and teardown, so it is easy to tell if eveything ran (died after test # 12).

It's not really that aspect that causes the problem. Indeed Perl has a couple of testing frameworks that use the xUnit model (Test::Unit and my Test::Class).

The issue is that in frameworks like JUnit, TestNG, etc. the test runner and the tests are running in the same process. If the tests exit for an unexpected reason then nothing runs - so hard crashes of the system are trivial to detect (hey - my test suite has crashed!)

In Perl the test runner (Test::Harness) and the tests (the *.t files) run in separate processes.

This is good in many ways, because it makes our tests more flexible (e.g. we can run tests in different programming languages) and has the advantage that a hard crash will not stop all the other tests running.

The disadvantage is that the test runner has to figure out for itself whether that test script exited because all the tests had run, or whether it exited because something unexpected happened.

One way it does this is via the fixed plan. Double checking that the number of tests that were expected to run actually ran.

It seems very un-lazy to have to run around updating that number all the time, counting properties manually and so on. I do not see the win.

The win is that we have a more flexible testing framework by splitting the test runner and the test producer up.

The cost is that detecting unexpected failures is harder. There's been some conversation on the perl-qa list recently on just this topic, with some ideas that might make the 'no_plan' option even safer.