I think that increasingly we're seeing people running distribution test suites in parallel by setting the environment variable HARNESS_OPTIONS=j<n> or by executing prove -j9. chromatic discusses it in his blog here: Parallelism in Test Suites. In that post, and in the discussion that follows, module authors are encouraged to be mindful of parallelism in testing so that in the vast majority of the cases where parallelism is possible module tests are designed to facilitate that capability. An example is given where two test scripts depend on the same clean slate, and where running in parallel they stomp on each other's sandbox. In most such cases, fixing the assumptions or creating independent resources for each test script is a reasonable fix.

In a followup to the same post, aristotle makes the following statement:

Making a test suite not break under parallelism doesn’t necessitate making it run in parallel. However I believe something like say 95% of tests on CPAN will already run fine in parallel with no further ado, and of the rest, easily the majority will be very simple to fix.

In the quasi-infinitesimal remainder of cases, sure, if the effort is not worth it, just forcibly serialise the tests and move on.

I expect a push to test parallelism to require little housekeeping effort all told. There just needs to be a reliable pressure that steers the CPAN towards it.

This node is about the quasi-infinitesimal remainder of cases. syphilis maintains Inline::C, and I maintain Inline::CPP. Those modules both depend on a C/C++ compiler to do much of the heavy lifting. I don't know about all popular C/C++ compilers, but I have found that gcc doesn't seem to support parallel compiling. If two test scripts cause the C/C++ compiler to be invoked at the same time, we get a test failure. I don't think our chances are strong for getting that fixed. So the question arises, how do we, as aristotle suggests, "just forcibly serialize the tests and move on"? syphilis and I have been discussing this issue between us, and could use some enlightenment.

One thought is for each test script to set $ENV{HARNESS_OPTIONS} by stripping out the j<n> flag, and then cleaning up any ':' mess in the case of multiple flags having been set. But that won't work because by the time a test script is executed it will already be running in parallel with others; it's too late to affect the harness.

My original thought was there might be a way for Makefile.PL to cause make test to override any HARNESS_OPTIONS setting, but I'm at a loss as to how to accomplish this.

Another approach might be for Makefile.PL to detect the HARNESS_OPTIONS flags, and complain loudly before dieing. At least then someone installing a module will know why he's getting a failure (otherwise, the failures can be pretty opaque).

Does anyone know how best to deal with this sort of situation?

Other resources: Test::Harness, Controlling Test Parallelism with Prove.