Beefy Boxes and Bandwidth Generously Provided by pair Networks
Pathologically Eclectic Rubbish Lister
 
PerlMonks  

OT: How do you test without knowing the results?

by Anonymous Monk
on Aug 04, 2005 at 15:40 UTC ( #480881=note: print w/ replies, xml ) Need Help??


in reply to When test-driven development just won't do

A related question: what do you do when the test results are themselves unknown? For sufficiently complex problems, if we could do the calculation by hand, we wouldn't need a computer.

For example, I once worked doing runway analysis for cargo jets: it took a group of our programmers two hours to hand-check a single number on a single page. There were perhaps a 100 numbers like that on each page, and hundreds of pages for a single manual, for a single type of airplane, for a single client. Granted, it could be hand checked faster than two hours: the two hour figure was largely because the programmers were unfamiliar with all the charts and algorithms they had to wade through, but the problem is by no means simple; there is a lot of room for error at multiple stages, and the method remains prohibitively slow.

We were provided with a binary program from a formally trained performance engineer who we had contracted to do these calculations correctly, but we had to take those numbers, and format them and present them on paper to the pilots who flew the planes.

Technically, from a legal standpoint, proving the correctness of the numbers we provided wasn't our job. On the over hand, both our company's reputation and the physical saftey of the pilot and crew were on the line. That essentially made it our problem in practice. In theory, our documents were just a "guideline" to the pilot: but in practice, the pilot needed those numbers to comply with both legal and safety requirements. If he knew what those numbers should be, he wouldn't need our product at all.

I eventually quit that job because I couldn't reconcile a way to prove the correctness of the numbers we were generating. These numbers were important to the flight of the plane (how much weight to put on it under given flight conditions, at what speed should the pilot lift off the runway, at what speed should he abort takeoff, at what height the pilot should stop climbing, and level off the plane to avoid hitting obstacles in case of an engine failure). While a good pilot could probably compensate for any gross errors, I decided that I didn't want to risk contributing to a flight disaster.

Can anyone think of a better testing approach that I could have used? I used to sanity check the most strangest airport destinations I could think of (ones with mountains, or at strange elevations, or with really short runways), and look for errors so obvious that even I could find them, but lacking institutional support (eg. a testing department, code reviews, someone else testing my code, etc.), and formal training (in both testing methods and aircraft performance engineering, etc), I really couldn't find a way to do my job to the level I felt I needed to.

As a result, I've gained an appreciation for testing, and formal correctness analysis. I'm always curious about how people improve their testing methodology, and especially how they convice management to let them do it. And today I work on things like billing systems, where nothing falls out of the sky if a bill goes out wrong.
--
Ytrew Q. Uiop


Comment on OT: How do you test without knowing the results?
Re: OT: How do you test without knowing the results?
by BrowserUk (Pope) on Aug 04, 2005 at 16:51 UTC

    For saftey critical systems, producing such data should be done by at least two completely independant programs, generated in clean room conditions by two completely different teams working from a theoretically proved, or engineering(ly) determined specification.

    The testing is done by comparing the output of the two systems and investigating any anomolies.

    This is a similar technique to that used by fly-by-wire systems on commercial aircraft. Three separate computers, often with different cpus to detect things like the Pentium floating-point bug, run different software written by different teams to the same spec. The independant computers are supplied the same information and perform the same computations, and another independant computer verifies their results against each other. If one of the computers produces different results from the other two, then the control computer will disregard that system's output and go with the other two. If one of them starts to produce consistantly different results, then it probably gets shut down.

    What happens if all three produce substantially different results? Panic I guess.


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    Lingua non convalesco, consenesco et abolesco. -- Rule 1 has a caveat! -- Who broke the cabal?
    "Science is about questioning the status quo. Questioning authority".
    The "good enough" maybe good enough for the now, and perfection maybe unobtainable, but that should not preclude us from striving for perfection, when time, circumstance or desire allow.
      For saftey critical systems, producing such data should be done by at least two completely independant programs, generated in clean room conditions by two completely different teams working from a theoretically proved, or engineering(ly) determined specification.

      The testing is done by comparing the output of the two systems and investigating any anomolies.

      Yup. Parallel testing is good, but it wasn't something I could do by myself. It requires the expertise of a second team of aircraft performance engineers to interpret the charts and tables correctly. Those guys don't work for cheap, and my company wasn't willing to pay for that sort of thing. Testing is much easier when you've got resources to spend on it.

      In general, the real problem with my company was that it wasn't willing to pay to do things right; when I arrived, I was told we badly needed a testing department, and expected to create one. When I left 2 1/2 years later, I was still testing my own code, there was only one Q/A manager, with two guys under him. A few months later I learned they'ld fired the Q/A manager... I wasn't sorry I left.

      I guess my original problem wasn't really solvable: "how do you do a good job of testing, with no support from management?" I think the answer is: "You don't."
      --
      Ytrew Q. Uiop

Re: OT: How do you test without knowing the results?
by Anonymous Monk on Aug 04, 2005 at 17:01 UTC
    Les Hatton has some interesting papers you might like...
    My work in computer science has primarily been in the field of software failure. Much of it has involved the design and execution of experiments to attempt to determine the cause and hence to reduce the occurrence of failure in software systems.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://480881]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others chanting in the Monastery: (5)
As of 2014-08-31 06:21 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls