It didn't happen every time and it didn't happen on the same database call every time. The test itself didn't give me any ideas. Its job is to run through all the modules, find the ones that implement a standardized interface and test that interface. About half our modules implement this interface and they all passed the tests, when the test script got that far.

I searched the web and the docs, but found no other mention of the error message. I posted to the DBI and MySQL mailing-lists, but got nothing back. Finally I found something that made the problem go away; I put in a call to $dbh->ping to check the DBI object before each time the $dbh is returned. I didn't think this was much of a fix, but it did avoid the error.

Weeks later a co-worker mentioned seeing a similar problem report on the DBI mailing-list. I rushed to check it out and found a reply mentioning fork() as possible culprit. As every good DBI geek knows you can't use a connected DBI object across a fork(), and the error message I got was a common symptom.

Eureka! The way the test works is by loading every module, calling $module->can('foo') and then proceeding to test the $module->foo() call. But in loading every module it happens to load one that does:

  BEGIN { Proc::Daemon::Init }

That call fork()s! Sure enough, as soon as I changed the test to skip loading this module the error stopped happening, even without the $dbh->ping() check.

I find this episode interesting for two reasons:

1. Sometimes the bug is the test itself. I might have found it earlier if I'd thought carefully about what made this test different from all other tests.

2. The proverb "only perl can parse Perl" is an important one. It's more than just a slander of Perl syntax. It also reminds us that parsing a Perl script or module also means executing it. You can't just go loading Perl code unless you're certain you know it won't do something while loading.

-sam

PS: Props to Kalin Kozhuharov for the post to dbi-users that put me on the right track.