Testing: To Mock or Not to Mock?

[agianni]

I'm working with some co-workers to develop some best practices for unit test writing and we've run into at least one conflict over what that best practice should be. While there are a number of unresolved questions, the most contentious question was of the degree of isolation that is appropriate in tests. I am generally of the school of thought that isolation is always disable, so I make a lot of use of Test::MockModule and Test::MockObject.

Here's a simple example of some code I might write and the test I would write for it:

use strict;
use warnings;

use Test::More tests => 2;
use Test::MockModule;

my $mock_simple = Test::MockModule->new( 'Simple', no_auto => 1 );
$mock_simple->mock( foo => 1 );
$mock_simple->mock( bar => 'bar' );
$mock_simple->mock( baz => 'baz' );

my $simple = bless {}, 'Simple';

is( $simple->foobar, 'bar', 'got bar when foo is true' );

$mock_simple->mock( foo => 0 );
is( $simple->foobar, 'baz', 'got baz when foo is false' );

####################################################################

package Simple;

sub foobar{
   my $self = shift;

   if ( $self->foo() ){
     return $self->bar();
   }
   else{
     return $self->baz();
   }
}

I find this useful because I don't have to care about what bar and baz actually do, especially if there is a lot of indirection involved in their implementation; i.e. I don't want to have to dig through three or four (or seven) layers of methods to figure out what to expect for the test.

However, there is a contingent at the office that thinks that setting up the object as implemented is a better option when reasonable. For example, if this was a CGI script and the foo, bar and baz methods were looking at CGI params to determine their return values, I would actually set those CGI params rather than mocking up the methods. The argument here is that this method provides at least a modicum of integration testing between the methods and is likely to uncover errors introduced by interface changes, intentional or otherwise.

What do y'all think? If isolation is generally desirable, are there other kinds of tests that we can use to try to catch other kinds of errors? We do a limited amount of testing with WWW::Mechnize. We actually do less than we used to since we've started taking unit test coverage more seriously and we have no standard as to what we test with mech tests.

[Old_Gray_Bear]

I come down on both sides of the question. I mock-out a lot of stuff when building my unit-tests. The purpose of my unit tests is to verify the *my* code is doing the right thing, right? I can just take the documented results of those External (to my code) Bits and replace them with stubs, right? Sort out my bugs and get on with it.

But then I go back and add an integration test(or tests) that set up the appropriate environment for the External Bits and validate that the combination of my code and the External dependencies all work as advertised. Both sets of tests have to pass before I have a 'Clean Run'(tm) of my test-harness.

In the process of setting up the integration test, I may find that External Bit phoo has a dependency on another External Bit baht, so I mock that dependency and roll forward writing my test.

Once I have a clean unit+integration test, my level 1 test harness; I build build a level 2 integration test, replacing the mocked baht dependency with baht itself and write the integration tests for that. Rinse and repeat, until I have worked myself up to the top of the stack.

Yes it is a LOT of work. But, the first time that your test-harness fails because someone made a 'minor change' change four levels up the stack -- it's worth every hard fought second. And that's the real utility of a Test-Harness. When Someone makes a change, I get early warning of any issues before the Code reaches the User. Testing in depth and nightly cycles are the way to go.

[agianni]

I agree that exhaustive testing along those lines would be ideal, but it's not usually a reasonable option. My current project includes over 40k lines of Perl code and nearly that many lines of tests and I can't imagine how much more time it would take to implement full-stack integration testing.

I'm curious, though, about the value of integration testing to full depth. For example, if I have some code where sub A calls sub B, which in turn calls sub C, why do I need to test the full integration of these? Rather, why can't I just test the integration between A and B and the integration between B and C and suggest that I have testing the integration between A and C transitively? Might there really be situations where that would not suffice?

That said, backing up your mocking at least one level for the sake of integration does make a lot of sense, and I hadn't thought of it in such an specific manner before. Thanks for your input!

[dragonchild]

I mock everything. I even maintain DBD::Mock. The only thing about mocks is that you have to test your tire on a car against the road at some point or another. So, while I have dozens of SQL statements in DBD::Mock sessions, those statements had to come from somewhere and have to be verified against that somewhere when they change.

Remember - a test suite does two things. It allows you to exercise the API while developing (Test-Driven Development) and it verifies that changes don't break previously-working code. Mocks are much better at doing the latter than the former. Use them where they make sense.

[agianni]

Am I seriously getting warnocked on this? Look, I've changed the title to better express what my question is about :) Anybody have any advice?