Beefy Boxes and Bandwidth Generously Provided by pair Networks
Perl Monk, Perl Meditation
 
PerlMonks  

Re: Regression testing with dependencies

by PodMaster (Abbot)
on Aug 22, 2003 at 09:27 UTC ( #285725=note: print w/ replies, xml ) Need Help??


in reply to Regression testing with dependencies

comment tests which may fail due to known resource unavailability
No no no, skip those tests when that resource is unavailable ;) As a cpan-tester, I often see tests go to hell during automated testing because no user/pass/host has been provided (some defaults assumed) and a database connection can't be established. In such cases, the tests in question should be skipped.

MJD says "you can't just make shit up and expect the computer to know what you mean, retardo!"
I run a Win32 PPM repository for perl 5.6.x and 5.8.x -- I take requests (README).
** The third rule of perl club is a statement of fact: pod is sexy.


Comment on Re: Regression testing with dependencies
Re: Regression testing with dependencies
by ferrency (Deacon) on Sep 04, 2003 at 15:39 UTC
    Skipping the tests completely is a commonly used technique here as well. Unfortunately, that has the unwanted side effect of leaving a lot of code without tests :) (it's the problem I'm trying to solve)

    When you're writing a module whose entire existence is intended to connect to some third party's server using a proprietary networking protocol, you can't very well skip every test, because then you're not testing anything at all.

    Writing a fake server or dummy module to connect to also doesn't help, because then you're testing against how you think the protocol/server works, not how it actually works. You're making the same assumptions you made when you wrote the module you're testing.

    It's roughly equivalent to doing this:

    sub protocol_thing1 { return "thing1"; # protocol-dependant Magic String } # And a corresponding test: use Test::More tests => 1; ok protocol_thing1() eq "thing1"; # Gee, that was fun! # The problem is, the server throws an error unless # you use "Thing 1".
    As you may have guessed, No, I haven't found an adequate answer to my concerns yet :)

    Alan

      You're making the same assumptions you made when you wrote the module you're testing.

      One way around this is to do what large corporations do - have someone else write the tests. The developer provides the tester with the design specifications and nothing else. In fact, it's better if the developer and tester never speak of the specifications at all, to avoid contamination. Then, the tester writes tests to exercise the specification, not the component.

      That last is so important, I'll say it again -
      The tester writes tests to exercise the specification, not the component.

      Remember - the specification is everything. The component implements it and the tester tests against it. It is your whole world.

      Component testing has another name - unit-testing. Testing the implementation against the specification is something that the developer cannot do.

      Now, this doesn't mean you have to have an entire testing team. At one position, we did cross-testing. I develop something, then hand another developer the specification I developed against. Based solely on that, the other deveveloper tests my implementation. You'd be surprised how many bugs are found with just this simple solution. Only when the cross-tester ok's the implementation is the test team even notified that the specification has been implemented and is ready for their testing.

      Automated testing, imho, should be built from the test-team's tests. When I worked as a test-tool supporter for Motorola, the developers did unit- and cross-testing. (They may have also done some regression testing, but I wasn't aware of that.) It was then handed to the test team. The test team did both regression and integration testing. Once that was completed, a subset of the integratration tests was chosen and added to the regression test suite. We also had a regression test team that was involved in doing the complete regression suite. (When I was there over two years ago, the regression suite for any version of Motorola's BTS software exceeded 10k tests. A full automated run took almost 80 hours. I'm sure it's larger now.)

      ------
      We are the carpenters and bricklayers of the Information Age.

      The idea is a little like C++ templates, except not quite so brain-meltingly complicated. -- TheDamian, Exegesis 6

      Please remember that I'm crufty and crochety. All opinions are purely mine and all code is untested, unless otherwise specified.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://285725]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others making s'mores by the fire in the courtyard of the Monastery: (4)
As of 2014-09-21 11:46 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    How do you remember the number of days in each month?











    Results (168 votes), past polls