Beefy Boxes and Bandwidth Generously Provided by pair Networks
Clear questions and runnable code
get the best and fastest answer
 
PerlMonks  

Inheriting Tests and Other Test Design Issues

by Ovid (Cardinal)
on Sep 26, 2003 at 23:40 UTC ( #294571=perlmeditation: print w/ replies, xml ) Need Help??

We had a strange debate at work today. Basically, the question was whether or not it was appropriate to inherit tests. I argued that it was. If I write a subclass, I don't want to rewrite all of the tests. I just want to test what's in the subclass and possibly inherit the tests from the superclass.

The counter-argument, provided by another programmer, is that by failing to explicitly test superclass behavior in the subclass tests, we're losing information. For example, let's say I have a superclass that, amongst other things, formats numbers to have a comma every three digits. My subclasses might change some behavior, but in this example, never change the number format behavior and thus have no tests for it. The problem arises when the super class number formatter is changed to have the comma every four digits, but the subclasses don't have tests for this and thus we're altering subclass behavior without testing it.

Personally, if I have to change a super class, I don't want to find everything that might subclass from it and update all of the tests. I want to run the tests and just fix the ones that break. Copying superclass tests over and over to subclass tests strikes me as a bad idea, but I have to admit that the counter-argument was interesting.

Have you ever used Test::Class or something similar to subclass tests? Do you think it's a good or bad idea? I'm also curious about whether or not you think that good software design considerations are different for tests. Frankly, I see a lot of test code that appears poorly written (cut and paste, mostly), but even bad tests are better than no tests, right?

Cheers,
Ovid

New address of my CGI Course.

Comment on Inheriting Tests and Other Test Design Issues
Re: Inheriting Tests and Other Test Design Issues
by Zaxo (Archbishop) on Sep 27, 2003 at 00:31 UTC

    I think you will need to be fairly explicit about which tests you import. You might fail superclass tests of an overridden function, or simple ctor tests which may have the namespace hard coded, not anticipating this kind of use.

    Do you have a mechanism to obtain the superclass's tests? Most module installations don't install their test suite to the system, but only run it from the build directory, blib'ed. If installed from CPAN, the source tarball is around in somebody's .cpan directory, but another user may not be able to see it.

    After Compline,
    Zaxo

Re: Inheriting Tests and Other Test Design Issues
by BrowserUk (Pope) on Sep 27, 2003 at 02:11 UTC

    I'm with you on this one.

    • Each class should unit test it's own behaviour against it's desgn spec./ contract.

      The example you cited of changing functionality (123,456 => 12,3456) is a re-specification of the design or contract and probably shouldn't be done, or at very least, would require a very clear notation in the upgrade documentation.

      It would almost certainly be best if any such change to the published API would be handled by adding a new method rather than altering the behaviour of the existing one. Alternatively, making the changed behaviour selectable via a configuration or instanciation parameter may be acceptable.

    • User's of a class should test their own behaviour and trust the design spec./ contract.

      To do otherwise is to invalidate the whole purpose of the contract. The essence of which is that a module/class/library contracts to return a given set of outputs when provided with a specified range of inputs.

    It would also logically lead to a maze of dark twisty tests, where every module would start testing that substr counted from 0 not 1 and that * really knew how to multipy two numbers. Then, what about the CPU? I mean, there was that early pentium math bug, maybe we should all be testing that our math co-processors aren't similarly effected:)


    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
    If I understand your problem, I can solve it! Of course, the same can be said for you.

      The example you cited of changing functionality (123,456 => 12,3456) is a re-specification of the design or contract and probably shouldn't be done, or at very least, would require a very clear notation in the upgrade documentation.

      Yes, but shouldn't you have tests checking that your superclass didn't change its specification? Shouldn't that be very important tests?

      It would also logically lead to a maze of dark twisty tests, where every module would start testing that substr counted from 0 not 1 and that * really knew how to multipy two numbers.

      No, because your module isn't providing substr or multiplication functionality. It isn't subclassing the functionality of the Perl runtime environment. But if you have a class that's subclassing a class that provides number formatting, then your class is providing number formatting. That that happens through inheritance isn't something the users need to know (encapsulating of the implementation).

      It's the same if you buy a Ford. You'd expect that Ford checks that the tires stay on the wheels when going 100km/h, and don't assume that they delegate that to Goodyear (or whatever brand they use). Goodyear does its test, but Ford should do as well.

      Abigail

        Yes, but shouldn't you have tests checking that your superclass didn't change its specification? Shouldn't that be very important tests?

        Yes. You should have a test suite for validating the functionality of your superclass, but not as a part of the unit tests of a subclass!

        If you were producing half-a-dozen subclasses of a superclass does it make sense to test the superclass functionality in the unit tests of all of them? What if it's a dozen, or two dozen?

        The owners/ authors/ suppliers of the superclass should maintain and use a Functional Verification suite for that class. You, as the user/ purchaser of the class should have an acceptance test suite that verifies the functionality.

        If logic and good relations prevail, then these may be the same test suite -- but that's neither essential nor always possible -- but this type of testing should only be run when the superclass is upgraded, not when testing a change to a subclass of it.

        Tight coupling between unit tests and the unit is essential, but equally essential is that units (modules/ classes) and be loosely coupled, that includes their testing. A unit test failure should directly indicate a failure in that unit, and not upstream from it. Upstream failures should have been detected before the upstream code is accepted.


        No, because your module isn't providing substr or multiplication functionality. It isn't subclassing the functionality of the Perl runtime environment.

        Nor is my module implementing the functionality of the superclass -- its just using it, just as it uses the perl runtime, c-runtime, OS systems calls -- or passing it through.

        Unit tests of my units functionality, should automatically show up any failures in the superclass, where they affect that functionality.

        Any tests aimed solely at verifying the functionality of the superclass are either

        • duplicating testing already performed there,
        • or testing the effectiveness of the functional verification or acceptance tests.

        This should not be necessary, and is undesirable, as all it does is increase the development/maintainance cycle and ultimately increases costs. Duplicated testing doesn't improve anything. If you don't have faith in the testing of the superclass, put the effort in to improving it, not duplicating it.

        Test thoroughly, but only test once! Or rather, in one place.

        I've got some great (but long) horror stories of corperating testing methods, and how more isn't better if it's more tests of the same thing.


        But if you have a class that's subclassing a class that provides number formatting, then your class is providing number formatting.

        If the number formatting is used internally by my module, then my unit testing should show up any disparities between the specified and actual returns I get from it without resorting to tests specifically aimed at testing the superclass.

        If my module is passing the superclass functionality through without overriding it, then any testing I did of that would either be testing the inheritance mechanisms -- which would be like testing substr -- or it would be duplicating testing that is (should) already being performed by the superclass unit or my Acceptance testing of the superclass.


        That that happens through inheritance isn't something the users need to know (encapsulating of the implementation).

        Agreed, but that doesn't enter into the argument. The users of my module, (and by proxy the superclass) should have their own FV or Acceptance tests for my module. If we can agree to share that between us, so well and good, but if the superclass has a bug or failure to meet spec. that isn't detected by my unit testing or my users acceptance testing of my module, then it is irrelevant or those tests are flawed and should be improved.

        It's not a case that anything should go untested, it's just that there is no benefit in testing stuff twice. Placing tests in the right place not only minimises the amount of testing done, and the costs involved in doing it, it also means that testing times are shorter, which encourages them to be used more frequently at each particular level, which improves overall throughput and quality.


        It's the same if you buy a Ford. You'd expect that Ford checks that the tires stay on the wheels when going 100km/h, and don't assume that they delegate that to Goodyear (or whatever brand they use). Goodyear does its test, but Ford should do as well.

        Nice (or perhaps, pertinent) analogy :)

        Goodyear should test the construction of their tyres 1 and ensure they live up to their specified rating SR/VR/HR etc. 2

        The wheel manufacturer (Ford or 3rd party) should ensure that their wheels correctly retain standard tyres -- not just the particular tyre chosen as standard equipment on one particular model that uses that wheel -- on the rim under all 'normal' circumstances.3

        Ford should

        • Pick (design) a suitable wheel for the vehicle.4
        • Pick a suitably rated tyre for their vehicle/ wheel combination, given it's performance characteristics, weight, probable modes of use etc.4
        • Test all three (plus all other standard equipment) in combination.5

        During this latter testing, it should not be necessary for Ford to perform lamination tests on the plys, or durability tests on the radial reinforcing, or test the compound for longevity or wet whether grip etc. This should all have been covered by the unit testing and be certified by the manufactures rating.

        In software terms, the tests described above fit roughly into these categories.

        1Unit testing.
        2Functional Verification.
        3Integration Testing.
        4Acceptance testing.
        5Systems testing.


        Examine what is said, not who speaks.
        "Efficiency is intelligent laziness." -David Dunham
        "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
        If I understand your problem, I can solve it! Of course, the same can be said for you.

You can't ever have the perfect classes, I guess...
by cbraga (Pilgrim) on Sep 27, 2003 at 05:16 UTC
    You write: The problem arises when the super class number formatter is changed to have the comma every four digits, but the subclasses don't have tests for this and thus we're altering subclass behavior without testing it.

    Well, I may be wrong, but it is my understanding that the subclasses trust the superclasses with respect to behaviour the subclasses don't define themselves. That is good, since the comma may have changed to four digits because the new protocol specification says so, and then all subclasses will have that changed automatically. I think that's a basic assumption of OO design.

    That is also bad because the superclasses are responsible for all their subclasses' good behaviour, and the programmer must, when changing how the superclass works, keep in mind all the subclasses he may be breaking.

    As projects grow large, that is both a blessing and a curse. The cure, well, is to keep strict documentation of what each class is supposed to do, and how. Which is also a pain in its own right.

    My take on the question at hand is that tests should be subclassed so that the inherited behaviour of the superclass will be preserved on the test. That's how OO works, IMO. If the subclass isn't supposed to change in behaviour when its superclass changes, then it's got no business in being inherited on the first place.

Re: Inheriting Tests and Other Test Design Issues
by tachyon (Chancellor) on Sep 27, 2003 at 13:17 UTC

    If would be my view that you personally should have unit tests for everything that is important to your system. If you subclass I would suggest you should

    1. First look at the code and the tests of the superclass (you are afterall throwing you hat in with it)
    2. Steal every last one of its tests, making only such modifications as are required to get them to run

    Why 'steal' the tests? At the end of the day your project depends on your subclass working. If you have a complete test suite for that subclass - specifically all the functionality you use - (as a specific part of your project) then you will immediately become aware of any issues (regardless of whether they relate to the superclass implementation OR your subclassing of it). And realistically you NEED to be aware of this, regardless of where the fault may lay.

    You are subclassing to save time/money on coding. I do not think this relieves you of the burden to thoroughly unit test your subclass which means testing all the functionality you are using - if you can steal tests from the superclass fine - but I suggest independent testing would be the more prudent approach. Sure standards change, but so do module authors. It is not unknown for dud distributions to make it onto CPAN. I would rather fix broken tests I duplicated from a superclass than have broken production code. Lesser of two evils to me.

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Re: Inheriting Tests and Other Test Design Issues
by Steve_p (Priest) on Sep 27, 2003 at 15:09 UTC

    I'm not sure that inheritance is the correct thing. In fact, I'm not really sure why you would use inheritance. You're tests should be testing behaviors. So, if two classes have the same behaviors, they should be tested together. For example, if I have a class A and subclass B and they both have a method foo and a->foo() will be equal to b->foo() in all cases, then they should be tested together in the same place.

    So, what I would do is something somewhat different from what you have above. For class A and subclass B, I would write:

    • A set of test cases for the common functionality of A and B.
    • A set of test cases for all the unique functionality in A. This might include constructors or methods overridden by B.
    • A set or class of test cases for all the unique functionality in B. This would include all the new functionality in B.

    Then, down the road, if I create a new subclass of A, I may have to refactor my test cases to cover all my bases, but it will keep me from repeating code.

Re: Inheriting Tests and Other Test Design Issues
by tilly (Archbishop) on Sep 27, 2003 at 20:47 UTC
    ..but even bad tests are better than no tests, right?

    Wrong.

    A bad API is not improved by a set of tests that nail down every aspect of its misbehaviour. Furthermore tests are code like any other, and good coding principles apply just as much.

    About your main question, I agree that inheriting the tests is better, and here is the argument that I would give back to the other programmer. Suppose that your subclass is written and you copy tests. If anyone now extends the superclass, then either subclasses have untested behaviour or else the implementer has to track down every subclass and cut-and-paste the behaviour. This is exactly the kind of cut-and-paste which is likely to break that object-oriented techniques are supposed to protect you from.

    However if your strategy is to inherit tests then both your present and future behaviour will get tested. True, it is possible to break your module by changing just the core module and changing its tests without notifying you. But this breakage is exactly the same risk that anyone using any module faces. No strategy can really protect you from rogue modifications, and attempting to do so by producing masses of redundant code loses productivity to what is really an education issue.

      If anyone now extends the superclass, then either subclasses have untested behaviour or else the implementer has to track down every subclass and cut-and-paste the behaviour.

      No, the complete behaviour of the superclass is to an extent immaterial to a widget that uses a subclass of it. As far as the widget's functionality is concerned what needs to happen is for the subclasser to test what is actually in use. That way MY tests fail if MY code is going to fail - this is ultimately what is required of tests. They let you know that some change somewhere has caused some new behaviour. The fact that a test exists means that that behaviour was probably important enough for you to write a test for in the first place and thus you should be concerned.

      masses of redundant code loses productivity

      A thorough test of subclass behaviour is not, and never will be, redundant. It simply ensures that the subclass continues to behave the way YOU expected when YOU wrote the code that uses it. If that changes (for whatever reason) you probably do have an issue you need to look at.

      cheers

      tachyon

      s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Re: Inheriting Tests and Other Test Design Issues
by dws (Chancellor) on Sep 27, 2003 at 23:06 UTC
    Basically, the question was whether or not it was appropriate to inherit tests. I argued that it was. If I write a subclass, I don't want to rewrite all of the tests.

    I look at it this way: I write objects because I want them to behave in particular ways. If I can save code by factoring common stuff out into superclasses, great, but at one conceptual level, that's just an organizational optimization. If I can get some behavior for free by subclassing an existing class, great. But again, that's an optimization (saving the time and effort it would take me to redo the equivalent work). What I still have is objects with behavior, and I want unit tests that cover that behavior. That means complete coverage of every class. Now, if it just so happens that there's some common testing that can be factored into a unit test superclass, great. But again, that's just an organizational optimization.

    I might end up with an inheritance hierarchy in my tests that doesn't match the inheritance hierarchy in my domain objects. Though I've factored domain behavior into abstract superclasses, I don't intentionally set out to write unit tests for those superclass. I might get the same effect by factoring unit tests, but I might not. But in all cases (in theory, at least :), my non-abstract classes have unit tests that cover the behavior I'm interested in.

    That's a slightly different way of looking at the world than the two options in your question allow, but it works well for me.

      I agree. If behaviors A, B, and C are provided by the superclass, then A, B, and C should be tested in the super-test. My subclass provides behavior D, thus I provide tests for behavior D, inheriting the tests for behaviors A, B, and C. If the superclass changes its behavior, it will change its tests. My testing of my subclass will inherit the changes in the tests. No problem.

      ------
      We are the carpenters and bricklayers of the Information Age.

      The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

      Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

        I agree. If behaviors A, B, and C are provided by the superclass, then A, B, and C should be tested in the super-test.

        That's not what I recommended. Please re-read what you think you're agreeing with.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://294571]
Approved by gjb
Front-paged by gjb
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2014-08-31 02:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls