Testing methodology, best practices and a pig in a hut.

[BrowserUk]

This isn't much of a meditation. More a reference to a couple of pieces of further reading on recent topics prompted by an enquiry.

Testing methodology

Anyone who's been around here for a while (and still bothers to read my posts), will know I have a distaste for Test::* suite of modules. I'm not opposed to testing. Indeed I spent a good part of my career doing pretty much nothing else and am 100% convinced of the need for and value of testing as an integral part of the software development process.

My argument is not about whether you test. It's about what you test. When. And how.

Being a vocal member of a usually silent majority(?), speaking out against practices advocated by the more vocal advocating minority, causes consternation: Why does he think he knows better than all those others?

The answer of course is, I don't. But that doesn't mean I'm wrong either. There is no one right way to do most anything. Some ways work better than others in certain environments; for particular teams or individuals; for particular project types or goals.

Just as getting from A to B may be better accomplished using a car, minivan, pick-up truck, bus, 18-wheeler, train or bike, depending upon the requirements of the journey involved. So RAD, Agile, XP, or waterfall development models will each suite some projects/teams/environments better than others.

If you are a someone who's programming skills are evolving through doing, you start out writing small scripts and debugging by trial and error. As your skills improve and your programs get more complex, you reach a point where you look for a better way. If there are a set of tools and frequent references to those tools and the underlying methodology they implement, you will take a look and give them a go. And if those tools and methodology are an improvement over your previous (non-)practices, you are likely to adopt them and become both practitioner and advocate.

Anything is better than nothing. And if you've only tried one thing, then that thing is going to be the anything that you use. But if you have experienced more than one methodology, whilst you may have an opinion as to which you prefer, it would be a rash man who elected to ignore the alternatives for every project he becomes involved in. Rasher still to recommend it, to the exclusion of all others, for projects that he has only the barest of knowledge of.

There is an old saying:

Good judgement comes from experience, and experience comes from bad judgement.

And I know of no field where that is more applicable than software development.

Best practices

I've also been known to express concerns that a certain set of "Best Practices" has had a detrimental affect upon the way the use of Perl's full syntactic power is perceived. These concerns relate not to the detailing of the best practices themselves, but to their codification as a set of rules enforceable through automation without a full understanding of their implications.

Try as I might, I can come up with no shorter explanation than the following. Drawn from a paper Why Software Quality Assurance Practices Become Evil! (pdf)

(UPDATE: The preceding link doesn't work without registration. However, going to here and clicking the link labelled "View Content Detail:" does))

that I highly commend to all those that see no merit in my apparently contrarian opinions. And those who believe that they are currently using the One True Way of software development and testing. It is only 11 pages, and just might give a few of you pause for thought.

So let me rephrase the title of this paper into a more verbose title: Why requirements management, software design, coding standards, inspections, unit test, integration test, system test, and management reviews become a source of discomfort and repulsion and are offensive. So why is this? Let’s start out by differentiating practices from principles.

Webster’s defines the noun practice to mean “actual performance or application” and uses as an example “ready to carry out practices that they advocated in principle.”

Webster’s defines the noun principle to mean “a comprehensive law, doctrine, or assumption.”

So practicing principles would mean “actual performance or application of a comprehensive law, doctrine, or assumption.”

My actual experience has led me to observe that it is a big mistake for the software industry to focus on the portability of practices from one software project to another. Practices, by their very definition, are specific to their application. We hear things like “best practices,” “standard practices,” “recommended practices,” “common practices,” and (my favorite) “best commercial practices” as if there is a finite set of them, and they work for all situations. Practices are the actions taken to accomplish something.

What if we focused on best principles instead of best practices? A principle is not application specific, therefore it is portable. A single principle can be instantiated into many different practices, each appropriate for specific situations. Principles could be instantiated into appropriate practices based on Risk, for instance.

The pig in a hut

For an explanation of that, you'll have to read the paper above :)

[stiller]

Common sense, unfortunatly, is not as common as we like to think.

And that is the problem with the application of best practices et al many places. But shops that don't even try to do something like best practices, code reviews etc, they don't really shine, do they? (Now, I don't hint at that being your agenda, I kind of agree with you).

Just take as an example how people go to extremes about Damian Conway's great book, Perl Best Practices. He goes out of his way trying to drive home the point that it is not a Bible. Yet a suprisingly large crowd hold it as dogma and use that either "for" it or against it.

I think we need to at least try to learn from each other, and code reviews would be great. Application of common sense would be really great too.

As for testing, yeah, a push for that would be great too. If only the bosses would even demand a proof that perl still is able to load a module, they might understand that they should ask for more interesting tests. :o)

cheers

Update
After actually reading the article I think I was on the right track above. See, the problem with both under- and overuse of best practices is that common sense is kept out of the picture. To a certain extent, best practises is meant to remedy temporal and accidental lack of briliance, but all too often they are the only thing keeping the house from falling apart, and then nothing is pretty.

Somebody else here asked about what this has to do with Test::*, and I wonder too. Care to elaborate?

d.s.

[chromatic]

My colleague Jim Shore just co-wrote a book called The Art of Agile Development. He had the same problem you describe -- how do you teach a novice how to become an expert capable of analyzing different development strategies and practices and choosing the most effective style for one particular team?

His idea for the book was insightful. He wrote the best advice he had, explained the places where it might not work, and said "Try it this way for a few months. When you understand it fully, including where it doesn't work perfectly for you, then change it so it does."

I like his idea, and try to give people the best advice I have. I don't see the value in being contrarian solely for the sake of making the world less clear.

[BrowserUk]

I don't see the value in being contrarian solely for the sake of making the world less clear.

Until that sentence, I had thought that my meditation might have met with your approval. At some level at least.

I am not contrarian for that reason. Indeed, I am not deliberately contrarian for any reason. If I am contrarian, and in this place and environment I certainly seem to be, it's not because I set out to be so.

I try, to the best of my ability, to make informed, logical, concious decisions about things, rather than just adopt the prevailing ethos. I do this in every sphere of life. And when I express my opinions, I try to explain the reasoning and logic that led to my decision. And I only express those opinions in response to direct questions asking for alternative views. Unlike some. And then, I explain my disatisfaction with ABC, and leave the questioner to make their own decision.

On the example that I think concerns you most, the Test::* suite exists and works (for many). But, if I suggested string eval for use in production code, I would be panned. But it is the basis of much of Test::*. If I suggested stealing both STDIN and STDERR from all your production programs, you'd be up in arms. If I suggested you verify the contents of your HTML files, by hard coding comparisons for every line, in sequence, you'd laugh me out of court. Yet for some modules, the test suite is fully a third of the code produced.

There are alternatives. The assert & trace macros in Smart::Comments] for example. Easily enabled. Combine with diff for automation.

As for making the world less clear. So many analogies spring to mind here, but I settle for one: The world would be more clear if there was just one god and everyone had to believe in it. Or none, and nobody was allowed to believe in one. Or many, and everybody had to believe in them all. In some places, at some times, all of those have been legislated. Would you advocate that? To make the world more clear.

[chromatic]

Yet for some modules, the test suite is fully a third of the code produced.

It's a big chunk of Pugs, Rakudo, Parrot, and Perl 5 as well.

Now I'm all for better abstractions and reducing the amount of work to get the right results correctly (I did extract Test::Builder to make this possible for testing, after all), but complaining that a certain unnamed hypothetical test suite seems big to you is just as silly as complaining that you used string eval in a certain unnamed hypothetical production program.

The interesting question to me is "Why?" I agree that a blanket prohibition on one or the other is unhelpful just as I agree that a blanket prohibition on the use of symbolic references or even indirect object notation is unhelpful and likely wrong. That doesn't mean I want to throw a novice head-first into the whole debate over all of subtleties of community idioms and best principles when all he or she wants to do is know why the program doesn't work.

[amarquis]

That paper was interesting, worth signing up to read.

I thought I was going to disagree, because it started with some bold words, but ended up at a fairly mild conclusion: That practices should be evaluated in the context of the business requirements of the project. I.e. look at the real cost of failures and balance that with the cost of practices designed to minimize failure.

Obviously, I can't speak for everybody here, but are there people that think otherwise? Preventing 90% of issues is fairly easy. Preventing 99% is hard. 99.9% is incredibly hard, and so on and so on. Obviously, you have to stop somewhere. And to decide where exactly to stop, you have to sit and think what the real cost of failure is. Will a small fraction of customers be driven away by the bug? Will embedded systems need to be recalled? I think that everybody goes through this "How good does it have to be, how much effort will it take to get there" evaluation when thinking about a project.

Talking specifically about the Test::* suite, I haven't seen zealotry here. I see comments like "Use of Test::Whatever is very beneficial when I use it." I've been taking that to mean "I use it where it makes sense, but not where it isn't an economic use of my time." I may be way off, but I don't think people are advocating cargo-culting use of Test::* modules into all projects.

[Anonymous Monk]

It's not about "a better way" or "the best way." It's about putting thought into the needs of a project and deciding what the best plan of attack is, not blindly doing either what you've done before or what everybody else is doing. If you say "We've always got to have extensive Test::* driven automated testing," you are burning a pig in a hut.

[kyle]

You can lead a monk to knowledge, but you can't make one think.

Some will think. Some will not. I personally think that a discussion of best practices/principles for testing or anything else should begin with encouragement to think. For those who have not the inclination to think, the capacity to think, or the experience to think clearly, a list of rules is better than no rules at all.

[kyle]

I can't tell what you're advocating, if anything.

I've heard it said that you can't define something by what it isn't. (It's true that a chair isn't a table, but "a chair isn't a table" isn't a definition.) I mention this because what I've read seems to imply "there is a better way than what is commonly advocated", yet there doesn't seem to be any explanation of what the better way is.

I think that different people work differently. They come to different methods based on their personal strengths and personal experiences.

I'm interested in reading about your experiences and the methods those experiences have taught you to value. I'm not so much interested in the advocacy of a better way that isn't presented.

[xdg]

I appreciate your attempt to clarify your position regarding Test::* -- and I think you make (by proxy) a very good point about the critical role of thinking about a problem rather than falling back on rote process.

However, what I would really like to see are some examples of the practices -- varied though they might be to each situation -- that you have used as an alternative.

The one CPAN module I found (Win32::Fmode) is rather spare on testing -- perhaps a symptom of Test::More frustration rather than an alternative.

I'd be interested in a meditation from you that lays out a few case studies along the lines of:

• Here is some code I was working on
• Here is some functionality I needed to test
• Here is how I decided to test this code

or

• Here is some code I inherited
• Here is the awful test code I inherited
• This is what I did to make the test code easier to work with

I think this would do a lot to educate us not just on your position -- which I think I understand better now and respect the core principles of -- but on your practices, which I'd genuinely like to learn from (without adopting slavishly, of course).

[BrowserUk]

I'll see what I can come up with.

[goibhniu]

Capabilitiy Maturity

I once worked with a company trying to achieve CMM level 3. Most at the company (developers and sysadmins alike) hated it. There was a feature in their process that I very much appreciated and have tried to keep in mind at other places I've worked.

They had a person whose role was titled "SQA". This person was responsible for reviews of every project at every project phase transition (already too much process for most people). A phase transition, eg, would be to get your project out of "Requirements" into "Design", or out of "Development" into "Testing". The thing was that at the beginning of the project, the project sponsor / project manager / tech lead would negotiate with SQA as to what deliverables would be part of the project.

They had an exhaustive list of standard deliverables (and a library of deliverable templates to get started on), but they also had specific lists based on project size. The base list for "small" projects was, well, smaller than the base list for "large" projects.

If you were a developer and wanted to sponsor a project with an order of magnitude less than 1 month of man hours, you could print out the "Small Projects Deliverables Checklist", take it to SQA and negotiate which deliverables you'd do in which phases of the project. The "Small" list was less than a page. And if that was too much, you could negotiate with SQA and put checkmarks next to the ones you were going to do and, if he agreed, you and he would sign it and that would be what you'd do. Then to get yuor project from one phase to the next, you'd show you'd done those. For "Small" project, even some phase transition reviews were not required.

If you were on a "large" project, the list of deliverables might be ten pages or so with various levels of more detailed requirements, design and functional specs, test plans and even meta-project documentation like team communications plans and GANNT charts, etc. Still, like the small projects, at the beginning of the project the project sponsor or manager (or both) and SQA would put check-marks next to the ones they would commit to do, and that became the contract for all the future phase transition reviews for the project.

People still hated all that process for all the heft that it (and it's waterfall model) implied. Also, the SQA guy got the reputation for being a hard-ass, as you might expect. Still, the cool thing that I took away was the flexibility to tailor the process to the correct size for the project. Most of the nay-sayers in the organization conveniently ignored that part of process in all their bad-mouthing.

Why Software Quality Assurance Practices Become Evil!

I liked the system proposed by Gregory Pope in the paper BrowserUK pointed us to. It reminds me of the SQA checklist I saw before, but adds a layer of sophistication in that it assesses the process based on risk, not just size (perhaps a "small" project might really threaten life and property and warrant a formal test plan, etc.).

Test::*

As to Test::*, I don't have enough experience good or bad to pass judgement. It did seem to me that Does anybody write tests first? attracted a lot of cargo cult responses. There was nothing in particular to suggest that amarquis was thinking of a high-risk or low-risk project, but he did seem to be talking about small projects growing organically. In that case, the Test::* options might be overkill, but I would still be talking to the imaginary SQA in my brain to negotiate how I would satisfy the principle of "preventing, detecting, and removing defects from the software system".

The last question I'd ask, however, is how many times do we re-ask the same question. Those that grab Test::* instinctively are saving some time by using tools they know and are familiar with. BrowserUK's response is that they may be spending more than they're saving depending on the situation. The company I worked for had a cool mechanism defined in their process that helped minimize the upfront analysis so that project teams could jump into the problem without spending alot of time on the meta-problem. I admit they didn't do everything perfectly and they had a long way to go, but I haven't worked for a company since then that was as intentional about pushing up their "Capability Maturity".

[zentara]

I'm not opposed to testing. Indeed I spent a good part of my career doing pretty much nothing else and am 100% convinced of the need for and value of testing as an integral part of the software development process.

Me too. My testing strategy is to post the script to Perlmonks, and see if I get complaints. :-)

[zby]

I fail to see how the whole argument connects to you distaste of the Test:: modules. Is there anything in particular about the Test:: suite of modules that you don't like - or do you want to say that no test suite will be appropriate in 100% cases?

[BrowserUk]

Is there anything in particular about the Test:: suite of modules that you don't like - or do you want to say that no test suite will be appropriate in 100% cases?

You've both missed my point, and made it exactly with your assumption that the only way to construct a test suite is to use Test::*.

[zby]

Sorry, but I don't see where I am assuming anything - I am just asking a question. You say:

Anyone who's been around here for a while (and still bothers to read my posts), will know I have a distaste for Test::* suite of modules

and I am asking what is it that you don't like about them. Is it something specific to that set of modules, or is it something along the lines that there cannot be any good set of test modules? Or maybe you are using it as a metaphor for testing in general?

[stiller]

I'd love to see you elaborate on alternative techniques. Testing is often seen as a chore, and in the shop I work in, the problem is a blatant lack of testing (especially automated). More productive testing strategies might help push testing among peers, and that would be a good thing.

Better ideas for more productive testing are always welcome to me, but I'm not sufficiently trained or smart enough to derive more productive testing strategies from what you've written so far. I do use Smart::Comments, which I think you've mentioned, a little, but probably not as effective as I could. Also, there must be more to your techniques than using that?

[tye]

We're sorry, you must be logged in to access this area.

Poo. Both of the bugmenot logins are no longer valid. Ugh, the registration process is rather intrusive. Even after I've registered, I've yet to be able to get to this supposed file. My browser just spins, apparently waiting.

Just FYI to any who might be interested in following your link but not that interested.

[amarquis]

Instead of following the link in Browser's node, once you are logged in search for the article title. Mine didn't load from the direct link either.

Edit: You don't even have to be logged in, either search from the main page or click Browser's updated link.

[tye]

A little birdy hinted and I found *mumble* kind enough to very quickly feed me the article (and not as PDF). I found the text unable to hold my attention so I was quite grateful for your very short summary of it. I did skip through it and see that it appears to have been drawn from rather extreme examples as well.

[BrowserUk]

I apologise that the link didn't work. That is the exactly url I used to download the pdf.

Clicked that link from this page. And I got to that page from google. I never had to register anywhere. I guess they must be use REFERER.

Again, I apologise.

[ysth]

"We're sorry, you must be logged in to access this area."

Not being able to read the paper you link to, I have only your words to respond to. Your argument seems based on "best practices" meaning nothing more than "best" plus "practices" and that "practices" aren't reusable in different settings (and so can't form a basis for software development methodology). How about assuming that when others say "best practices", they mean what you mean by "best priciples"? What is the "practical" :) impact of your argument then?

[Gavin]

Nice analogy "Pig in a hut"
I'm very fond of a bit of crackling myself!

Problem is the natives are still burning down the huts.

[dpuu]

My perception is that much of the problems associated with testing is that the act of testing is an explicit and deliberate (and necessary) violation of the DRY principle ("Don't Repeat Yourself"), also known as once-and-only-once.

The rational of DRY is that if you state a fact twice, then when you change it then you need to change it twice. This doubles the inertia inherent in the code, and thus slows down progress (defintion of interia from physics: resistance to change).

The majority of unit test that I've seen (obviously an infinitesimal minory of those that exist) tend to be poorly factored. It's not uncommon to see code along the lines of:

is sqrt(4), 2;
is sqrt(16), 4;
is sqrt(9), 3;

Which repeats 3 times the the facts that: a function to evaluate square roots exists, that this function is named "sqrt", that it takes a single numeric argument, and that it returns a numeric value! Change any of those facts and the whole set of tests needs to be revised! Moving to a data-driven approach can mitigate this problem somewhat, but never entirely eliminates it. It can't, because the basis of testing is to violate DRY.

In this example, the facts being repeated may seem somewhat trivial, but as a general rule, the duplication acts to magnify the impact of imperfect design decisions: the added inertia may cause people to delay a needed refactoring; and the refactoring may itself need to be staged to avoid a scenario where both the tests and the implementation change simultaneously (e.g. renaming the sqrt function, above, would cause you to change both the tests and the implementation: a purist would tell you to first add the new function to delegate to the old; then change the callers; then the tests; and finally replace the delegation with the original implementation. It's not surprising that some of us give in to the temptation to take shortcuts!

Ideally, the violation of DRY that is the purpose of testing should be realized in a form that is as different as possible from the implementation being tested. Embedding the information as static type systems to be checked by a compiler; or as DBA (design by contract) to be verified by proof tools tend to form stronger whitebox tests than the simple test-by-example that, from my limited perspective, seems all too common.