UnitTesting multi-threaded apps

[alain_desilets]

I am starting to use mutithreading in an application that is unit tested with PerlUnit. The problem is that part of the application uses a module called KinoSearch, which is not thread safe. The following page: http://perldoc.perl.org/threads.html#BUGS-AND-LIMITATIONS says that you can get around the issue by loading the non-thread-safe module only at a time when you know that no new threads won't be spawned. This is completely uinrealistic when you are running a bunch of unit tests with PerlUnit. I have several unit tests which create instances of various classes and exercise them in certain ways. Some of these classes will be spawning new threads, and some won't. PerlUnit gives me no control whatsoever on the order in which tests are being carried out. And even if it did, I have many different tests which spawn new threads, so it's pretty hard to determine a point where no new threads will be created. How do people deal with this issue? Thx Alain Désilets

[BrowserUk]

If your testing methodology is not capable of allowing you to test your code, you have to question that methodology.

Once you start allowing the testing methodology to dictate what code you can and cannot write, you are on a very slippery slope.

[eyepopslikeamosquito]

It's well known that unit testing and multi-threading don't mix very well. This is still an active area of research. I touched on this topic recently in Nobody Expects the Agile Imposition (Part VI): Architecture (see the "Unit Testing Concurrent Code" and "Testing Concurrent Software References" sections).

In practice, I usually unit-test my software components in a single threaded environment only. For flushing out concurrency bugs in multi-threaded code, the most effective (if crude) technique I've found is good tracing code at just the right places, combined with understanding and reasoning about the multi-threaded code and performing experiments. One especially useful experiment is to add "jiggle points" at critical places in your concurrent code and have the jiggle point either do nothing, yield, or sleep for a short interval. There are more sophisticated tools available, for example IBM's ConTest, that use this approach.

[alain_desilets]

You wrote:

It's well known that unit testing and multi-threading don't mix very well. This is still an active area of research. I touched on this topic recently in Nobody Expects the Agile Imposition (Part VI): Architecture (see the "Unit Testing Concurrent Code" and "Testing Concurrent Software References" sections).

So far, I haven't found it hard to write tests to verify the multi-threaded parts. For example, I have a class called JobQueue which acts as a shared object between a Boss thread and a Worker thread. I have been able to write tests to verify that two separate thereads can indeed use this class to communicate in a Boss-Worker mode.
My problem is that I can't run the JobQueueTest in the same run as some other tests that use non thread-safe modules.
I guess I could write a shell script that starts two processes, one to test the thread-safe bits, and one to test the non-thread safe bits.
You wrote:

In practice, I usually unit-test my software components in a single threaded environment only.

This of course, assumes that your program can run in both modes: multi-threaded and single threaded. This makes the design more complex, but I guess it's a good idea anyway, given that perl threads are still somewhat bleeding edge (in other words, it would good for me to make sure that I can fallback on single threading if it turns out that multi-threading is causing too many problems). I think it should be possible for me to do that.
You wrote:

For flushing out concurrency bugs in multi-threaded code, the most effective (if crude) technique I've found is good tracing code at just the right places, combined with understanding and reasoning about the multi-threaded code and performing experiments. One especially useful experiment is to add "jiggle points" at critical places in your concurrent code and have the jiggle point either do nothing, yield, or sleep for a short interval. There are more sophisticated tools available, for example IBM's ConTest, that use this approach.

Can you elaborate on the jiggle point technique?
Thx.

[eyepopslikeamosquito]

I am starting to use mutithreading in an application that is unit tested with PerlUnit.

As an aside, why are you using "PerlUnit"? Are you new to Perl? I often see this with programmers coming from other languages to Perl looking for the Perl equivalent of JUnit. The mainstream Perl QA community overwhelmingly uses the core Perl Test::More module, along with the prove command and CPAN Test::More-compatible modules. If you need something like JUnit, take a look at Test::Class.

[alain_desilets]

You wrote:

As an aside, why are you using "PerlUnit"? Are you new to Perl?

LOL! I have been using Perl since 1995! Also Java, Python and PHP.

I often see this with programmers coming from other languages to Perl looking for the Perl equivalent of JUnit. The mainstream Perl QA community overwhelmingly uses the core Perl Test::More module, along with the prove command and CPAN Test::More-compatible modules. If you need something like JUnit, take a look at Test::Class.

Well, I went with PerlUnit, because I had been using jUnit, pyUnit and phpUnit quite happily and wanted to use the xUnit equivalent for Perl.
I have 640 tests written in PerlUnit for my application, so I am not really keen to switch in this particular project, but I might consider other frameworks for future projects. Can you tell me a bit more why the Perl commmunity does not use PerlUnit? What are the advantages of these other frameworks you mention? Would any of them solve my multi-threading test problem?
BTW: I just looked briefly at Test::Class, and I am not sure it would work for me because it's not object oriented. And testing an object-oriented app with a non-object oriented test framework doesn't work well for the following reason:
Say you have a class hierarchy like this:

ParentClass
   ChildClass1
   ChildClass2

There is some behaviour that the two children inherit from their common parent. This behaviour needs to be tested of course. If your test framework is not OO, then you will need to repeat those same tests in the tests for both children. But if your test framework is OO, you can create a parallel hierarchy of tests cases:

ParentClassTest
   ChildClass1Test
   ChildClass2Test

The tests for the parent class behaviour are defined in ParentClassTest, and in tests for the children only need to test that part of the behaviour which is specific to them. This is a standard testing best-practice, and I have found it to be really useful. But it requires an OO test harness. Thx

[BrowserUk]

Unfortunately, the meaning of the term "unit testing" seems to have been lost, or supplanted.

Unit testing should (did) mean testing a 'unit' of an application, and was only one of several forms of testing. It nows seems to have taken on a different meaning entirely, and to the detriment of everyone.

In a multi-threaded application (or multi-processed) application, unit testing should not require any concurrency. Each thread procedure should be tested (and testable) in isolation. That is the very definition of 'unit testing'.

Once unit tests have been past, then you move to the next level of testing, 'Integeration tests'. Where you bring those units together and test that they integrate. Integration testing inevitably does not (often cannot) fit with the OK/Not OK pattern of results required by these unit testing frameworks.

But, as is often the case with silver bullet development paradigms, the counting of OKs & NOKs has become more important than what they simplistically represent, and so you arrive at the situation where people attempt to force fit all their phases of testing into the unit testing mould.

The fundamental issues of integration testing (and systems testing if anyone still does it), especially in concurrent systems, are such that they cannot (and should not) be forced into simplistic, static Yes/No tests. They need to consider the effects of non-determinism upon the interactions between units of concurrency. And that invariably requires running the integrated units together over time and monitoring both the content and timing of their interactions.

Doing this well, or even at all, with a fixed sequence of static binary truth tests simply isn't possible. Attempting to force fit intgration testing into the unit testing mould for the sole purpose of the statistics produced is counter productive in the extreme.

[eyepopslikeamosquito]

Can you tell me a bit more why the Perl commmunity does not use PerlUnit?

PerlUnit seems to be have been abandoned. To put that to the proof, you might like to ask your question on the PerlUnit mailing list and see what sort of response you get.

OTOH, the author of Test::Class is an active member of the Perl qa community, fixes bugs quickly and makes regular releases. Test::Class is more Perlish, works with other Test::Builder based modules and has excellent community support. This is all discussed in the Test::Class documentation.

As for why PerlUnit never seemed to gain traction in the Perl community, that is an interesting question. My guess is that it is mainly cultural: Perl has a long tradition of testing and qa, it worked well, folks were keen to further improve it and saw no reason to switch to an xUnit-based framework. Perl has a thriving qa community and continues to hold well-attended Perl QA hackathons. Also, Perl is a multi-paradigm language (in constrast to OO-only Java where JUnit rules). Notice that multi-paradigm C++ has similarly not embraced an xUnit framework the way Java and C# (NUnit) have.