Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re^2: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)

by fokat (Deacon)
on Oct 24, 2005 at 00:46 UTC ( #502354=note: print w/ replies, xml ) Need Help??


in reply to Re: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
in thread Neither system testing nor user acceptance testing is the repeat of unit testing (OT)

I mostly agree with you, dragonchild, except for this small piece (which does not deprive you of your well deserved ++):

unit-testing is still black-box testing

The blackness of the boxes you test with, are orthogonal to the phase of testing you are at. Seriously, you can base your cases on the documented behavior (what the code is supposed to do) and also add cases to stress the inner workings of the code you're looking at.

If you generalize, you'll see that you cannot insure you're achieving the desired level of test coverage, unless you peek under the hood and verify that the code is being excercised in the intended (or unintended, depending on who you ask) way.

The point is that you must try to use both, black box and white box, as long as you can. As you progress in the scope of your tests (ie, you move from unit testing to system testing also called integration testing), the number of cases / paths makes the white box approach too resource intensive.

Otherwise, you may have all the test cases you want, and still miss bugs because you did not see how the actual code did something in particular.

But even when white box can help provide more effective test cases, this does not mean you can forget about black box testing. If you rewrite a substantial part of the code, chances are your white-box test cases become less effective, because they may now be tickling different code, or in different ways. But your black-box tests will still be verifying that at least, the interface is working as expected.

Update: pg is right in the money (++ too): For unit testing, most of the time, it is a mixture of white box testing and black box testing (...)

Best regards

-lem, but some call me fokat


Comment on Re^2: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
Re^3: Neither system testing nor user acceptance testing is the repeat of unit testing (OT)
by dragonchild (Archbishop) on Oct 24, 2005 at 01:10 UTC
    Oooh. This is where I start to get a little antsy. In my extremely un-humble opinion, white-box testing is a CodeSmell. If you need to look at the code instead of the interface to determine what tests you need to run, your interface isn't correctly designed. Correct interface design and mocking will allow you to black-box all your unit-tests.

    That's a big statement, but I'm willing to be proven wrong. (Yes, that's a challenge.)


    My criteria for good software:
    1. Does it work?
    2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
      OK - in reponse to your challenge, I've got one word to say to you - Exception Flows ;-)

      Say you have a function/method, as part of your interface, that writes a file out somewhere. This isnt the main role of the function/method, its just part of the work it does for you. The normal flow is to open the file, write whatever, and close the file.

      But there is also code to deal with being unable to open the file, not all the data being written or the file not being able to be closed.

      How these failures get communicated back to the client can vary too - from not saying a word through to throwing an exception or even calling exit().

      My opinion is that the developer is required to setup test cases for each of these failure modes, and this requires a white box approach. Knowing that the code writes a file may not be inherently knowable from the interface (whether it should be is a different point), but only by looking at the code do you _know_ that it writes a file, and that it does different things for different failures. Even if the interface doco states that it writes a file, detailing the different failure paths is almost certainly not part of the interface design or doco (but may well be part of the interface _implementation_ doco).

      The function/method writer can test all the normal stuff just by reference to the interface, but setting up file-open failures and checking that the function does the right thing needs more detailed knowledge related directly to the code. This is where tools like Devel::Cover can help - exception flows pretty much stick out straight away as bits of code, related directly to your chosen implementation of the interface, that you have got to write test cases for.

      ...reality must take precedence over public relations, for nature cannot be fooled. - R P Feynmann

        Ouch. Any possible exception that might be thrown by you needs to be listed in your interface, because it's a possible "return value". In addition, you also need to list every single interface that you use, because your caller might need to look up their interfaces to see which exceptions they might throw.

        To recap, exceptions most definitely are part of the interface spec.


        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
      "... to determine what tests you need to run, your interface isn't correctly designed."

      There is a big hole in this logic. Testing should not and cannot depend on the correctness of anything else, regardless whether it is the interface design or the actual code. There is simply no dependency, we don't expect it, and we avoid it. The whole purpose of testing is to find faults with the application, from requirements to design, to coding.

      If we assume that the interface design is great, or depend on that, then the entire purpose of the testing is defeated right there.

      This node was taken out by the NodeReaper on Mon Oct 24 04:40:21 2005 GMT
      Reason: [ikegami]: Identical dup. Identified as dup by author.

      For more information on this node visit: this

      Since you've thrown down the gauntlet, let me pick it up.

      You present a false dilemma, suggesting that you only get to do one kind of test. And then you present an argument that black box unit tests should be more important.

      But the fact is that you can do both. And they pick up different kinds of errors. I fully agree that all of the tests of the basic interface should be motivated by the interface specification, not the implementation. That is, they should be black. But corner cases and special cases in the code are a common source of bugs, and that one may only reliably figure out where those cases actually are by staring at the implementation. (One may guess without peeking, but one only knows after looking.) So including white box unit tests to smoke out possible bugs in corner cases catches errors that black box testing may not.

      Therefore after you write your black box unit tests, there is value in adding white box tests as well. Given a positive return from doing the work, one should do it.

      This is also why white-box measurements such as coverage percentage (see Devel::Cover) have value.

      UPDATE: minor punctuation and added "white-box" to the last sentence to clarify my point.

        This is so funny. I feel that your argument against my point only ends up supporting it. "corner cases and special cases in the code" are very much a common source of bugs. However, I think that having corner/special cases is a CodeSmell. (Yes, I'm being idealistic here - there is no application whose code smells like roses.)

        Plus, imho, Devel::Cover is a measurement of the success of your black-box tests vis-a-vis your interface. If you have lines of code you don't have tests for, that's a problem that needs to be addressed. Granted, you might address is by looking at the code to see which interface components either haven't been tested or have been over-coded for. But, that doesn't mean the test suite has necesssarily become white-box. I often use coverage to both learn more about my interface and to remove code that shouldn't have been there in the first place. Still black-box.


        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?
      Correct interface design and mocking will allow you to black-box all your unit-tests. That's a big statement, but I'm willing to be proven wrong. (Yes, that's a challenge.)

      I basically agree, with a moderate number of caveats :-)

      • Legacy code. I'm going to carry on adding white-box unit tests to test free code that's not been designed with testing in mind.
      • Code clarity. Sometimes I think the code remains clearer with a couple of private methods than it does by factoring out a separate support class. If I think that I'll quite happily write white box tests if I need to tweak them.
      • Can TDD ever be black box? Since the next test is driven both by the spec and by your knowledge of the implementation you can argue that it's always white box. The kind of unit tests that TDD produce are very different from the sort of unit tests a pure specification based approach might take (see discussion on the TDD list for example).

        I think TDD is only done correctly if unit tests are black box. You write your tests and thereby define your interface, regardless of the implementation behind it. Once you get to the refactoring stage you (may) change your implementation to make your code better organized|more efficient|less redundant|etc. Your refactoring succeeds if you manage to change your implementation without breaking a test. If you then have code that's not being tested, you have one of two cases

        1. Your tests don't accurately describe your interface, because you forgot to test for an outcome that should be part of your spec.
        2. The untested code should be removed because it produces an outcome not included in the spec (because ideally your tests are the most thorough description of the spec)

        In the case of 1. you need to add a test, but you still want to test against the module interface, not against the particular implementation. This is a corner case where you're actually finding a better interface by coincidence while refactoring. If you're writing your tests correctly it shouldn't happen (though IRL it does).

        I do agree that it takes a special sort of mindset to properly separate your test-writing brain from your implementing brain, so that's a practical problem for all but the schizophrenics among us :-).

        In the case of retrofitting tests to code, one might divine the specification from the implementation. I will be doing this for PDF::Template over the next few weeks. However, the tests should still be testing the interface, not the code. I fully expect to find and fix several bugs over the next few weeks by thinking of things this way.

        In the case of private methods ... that's an interesting point. Now, you have a private interface that you're testing against. You write tests against the private interface, then tests against the public interface. Just because you didn't publish the private interface doesn't mean it's not a black-box test. The tests for the private method are against the privately-published interface of the private method.

        TDD tests shouldn't be driven by the implementation. Look at Pugs as a perfect example of this. Half the test-writers can't read Haskell to save their life, but it's still TDD.


        My criteria for good software:
        1. Does it work?
        2. Can someone else come in, make a change, and be reasonably certain no bugs were introduced?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://502354]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others studying the Monastery: (8)
As of 2014-09-23 20:35 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (241 votes), past polls