OpEd: Programming is not Team Sports

[sundialsvc4]

There is always another Silver Bullet.   Another guy or gal who writes a book to tell us how the reason why we experience problems in software development and deployment is because we’re doing it the Wrong Way.   Lately, the popular analogs for The Right Way seem to come from team sports and video games (as those who sit upon couches envision them to be) ... Agile, Scrum, Quake.

The common thread in each of these, it seems to me, is that “we are being paid to Write Code, therefore let us Write Code constantly.”   Let’s go from Something We Can Release to Something Else We Can Release every few hours.   Let’s put something new quickly in front of users’ faces to see if they like it, and they will surely say O What A Good Boy Am I.

I’d like to offer the observation that, when we build houses and bridges and other Big Heavy Things, we don’t do these things.   (Heck, even professional sports teams don’t actually do these things.)   What do they do?   They plan.   They plan absolutely everything, and they do it well in advance.   They present the customer with alternative designs on paper, and, before anyone is actually engaged to build anything at all, they can tell you that “We will need 1,381 feet of No. 14 electrical wire ... 768 #14 2-1/2" bolts with 1,000 pounds tensile strength; ... ”   Then, having done construction projects so many times before, they can lay out a schedule with contingencies and keep it, and, when they have done so, the building stands.

Given that the software which we build might be several times more expensive than one of those buildings, maybe we should pay much more attention to how they’re going about their entire process.

I think that too many developers derisively dismiss The Waterfall Model as being somehow oldy-moldy and out of fashion.   But perhaps we shouldn’t.   If we actually did seriously put our efforts into mapping out in detail what we were about to do before we actually did it, “actually assembling the thing” would become a very routine procedure that mostly championed accuracy ... like following a set of blueprints, or pouring a concrete slab in the proper way.   We would not be presenting the client with a partially constructed building when we asked him if he does or does not like that bedroom being at that particular location.   I submit that we tend to think that our particular mental madness plays by different rules, and that we invent all kinds of self-justifying strategies and strategems solely to justify that notion.   Maybe it ... doesn’t.   Maybe the people who make their living constructing very big physical things have had the right idea all along.

That’s my 2¢ ... What do you think?   What has your experience been, and why?   Please be specific ...

[moritz]

My father in law is a construction engineer. Once I told him that software projects often fail, take to long and/or are way over budget, and that I wished that we used the established engineering practises from his field. He looked at me with very wide eyes and asked me if I seriously believed that "his" embankment dam projects fared any better.

Note that those projects also have significant maintenance costs, often due to less-than-perfect upfront planning and execution.

A commonality between software and civil engineering projects is that planning ahead works well if and only if you know the environment, the problem domain and the requirement very well, and none of them are changing.

I often read programming job ads, and it seems to me that most of them focus greatly on the tools (you need to have 5 years of experience in $language and $framework and $webserver and HTML and CSS and Javascript and 20 years of scrum experience), but mostly ignore the problem domain. I think that is one of the big shortcomings of current "best practices" in the software business.

Explorations into unknown regimes are best done from the bottom up. You can best see that in long-term research projects that first explore the underlying effects before trying to build any devices from them. And any project outside of your domain knowledge is as new to you as a research project is to the scientists investigating it.

So, if you are a software architect who has already built five web browsers, and are now charged building the sixth, taking the waterfall approach might make some sense.

If you have built a browser, a word processor and a web server, building a new compiler is still a research project to you. You could chose your methodolgy accordingly.

[gsiems]

++ "...and none of them are changing."

This. It seems that it is the rare project that doesn't have changing requirements of some form.

When I was in college, lo these many years ago, I did house construction for a while. It was not uncommon to hear "Can you (move|add|remove) this (door|window|wall|other)?" in the course of things. As it's a lot easier to (move|add|remove) things before you've run the electrical, plumbing, mechanical, or closed up the walls than it is after-- getting the feedback a.s.a.p. is essential to minimizing cost/schedule.

Actually, now that I think of it, not everything WAS on the blueprints as the assumption seemed to be that the contractor would figure out the missing details as they went along.

[Anonymous Monk]

Actually, now that I think of it, not everything WAS on the blueprints as the assumption seemed to be that the contractor would figure out the missing details as they went along.

They call it "code", it dictates all your options, they have inspectors for it, and you have to know the "code" better than the inspectors, because invariably they'll try to deny permit approval for code violations

[DrHyde]

I actually think it's OK to ignore the problem domain in job ads for almost all programmers. When I was working on systems to support the manufacture of medical devices, and later on systems for support nurses and patients with managing drug use and stuff, I didn't have to know much (or, indeed, anything beyond what any well-educated layman would know) about the problem domain. I was just writing code to gather and process data, and I don't care whether your data is incidents of adverse reactions to a drug or the migration routes of the Greater European Spotted Bat. All I need is to be familiar with the tools that the employer uses, to be intelligent enough to learn the terminology and the basics of the problem domain, and to be confident enough to ask a medic or a bat expert questions when I don't understand the problem, or I think their specifications are vague etc.

[BrowserUk]

I think that too many developers derisively dismiss The Waterfall Model as being somehow oldy-moldy and out of fashion.

The Waterfall Model doesn't work. I'm not citing (just) my own opinion here, but rather that of the man, Dr. Winston W. Royce, that first described the waterfall model.

Yep! The guy that 'invented' the Waterfall Model, said it didn't work. Indeed, when he first described it in his 1970 paper:"Managing The Development Of Large Software Systems", he did so explicitly to show why it didn't work, and what needed to be done to correct the method's inherent, designed-in, causes of failure.

See this For a potted history of how the mis-citing of the Royce paper, lead to it getting accidentally adopted by the US military in the early 70's; and thence forth by many other organisations who blindly copied them; before being universally abandoned by all of them in the mid to late 80's because it failed so badly, so often.

“Those who fail to learn from history are doomed to repeat it.” -- Sir Winston Churchill

[jdporter]

Royce, in that paper, clearly isn't dismissing the waterfall model entirely; rather, he says it needs to be elaborated -- to be made more complex -- in order to work. Look at the last page, Figure 10. This illustrates the model he says works, and it's clearly a form of waterfall. So if one is going to cite the "wisdom" of Royce, one has to admit that he thought the 'waterfall', as it is understood and practiced today, does not inherently lead to failure.

"I'm willing to let Churchill have the credit." -- George Santayana

[BrowserUk]

Look at the last page, Figure 10. This illustrates the model he says works, and it's clearly a form of waterfall.

Sorry, but you are wrong.

The bit of fig.10 that looks like the waterfall model -- the bit to the right of the dashed line -- is simply a reiteration for contrast of fig.4, of which Royce says (in the text above fig.4):

I believe in this concept, but the implementation described above is risky and invites failure. The problem is illustrated in Figure 4. The testing phase which occurs at the end of the development cycle is the first event for which timing, storage, input/output transfers, etc., are experienced as distinguished from analyzed. These phenomena are not precisely analyzable. They are not the solutions to the standard partial differential equations of mathematical physics for instance. Yet if these phenomena fail to satisfy the various external constraints, then invariably a major redesign is required. A simple octal patch or redo of some isolated code will not fix these kinds of difficulties. The required design changes are likely to be so disruptive that the software requirements upon which the design is based and which provides the rationale for everything are violated. Either the requirements must be modified, or a substantial change in the design is required. In effect the development process has returned to the origin and one can expect up to a lO0-percent overrun in schedule and/or costs.

The bit of fig.10 to the left of the dashed line is his alternative designed to correct the flaws in the waterfall model, which with its feedback loops and iterative processes is clearly the antithesis of the (no going back; never throw anything away) waterfall model.

Instant analysis doesn't cut it here.

[chromatic]

Then, having done construction projects so many times before, they can lay out a schedule with contingencies and keep it....

I worked construction during college. Two of my close friends are in industrial construction here. (They both work quite a bit with Intel, so it's not small construction projects either.)

We've talked about the "software should be like construction!" fallacy a few times. We laugh about it, mostly because it's a rare job where you show up on the first day without a stack of change requests a foot high.

[aaron_baugher]

From Calvin and Hobbes:

CALVIN: How do they know the load limit on bridges, Dad?
DAD: They drive bigger and bigger trucks over the bridge until it breaks. Then they weigh the last truck and rebuild the bridge.

For what it's worth, I have built a (small) house in the "release early and often" sort of fashion, without any blueprint beyond some rough outlines on paper. Obviously there are disadvantages: sooner or later you'll realize you forgot to leave room for such-and-such, and have to adjust or redo something to compensate. You could end up spending a day breaking up concrete with a jackhammer because you forgot to put in the drain for your shower.

But there are advantages too. It's hard to know just what a room or an overhang is going to look like until you see it for real; blueprint drawings or even CAD mockups just aren't the same. If you're figuring it out as you go along -- and building with that in mind -- it can be easier to change your mind and make adjustments along the way when you have new or different ideas. You're not constrained to a blueprint because you don't have one.

There has to be a healthy balance, of course, between rigid pre-planning and unplanned chaos. But when it comes to software, I think there are valid reasons to lean more toward the chaotic end. It's a lot easier and cheaper to tear down a program and start over than it is to tear down a house and start over. And release-early-and-often gives you much faster feedback than you can get by doing your own testing. (How many times have you found a bug in a piece of commercial software and thought, "How could they possibly miss that?" That was a regular thing in C64 games back in the 80s, even from software houses with large teams of testers.) Faster release cycles mean faster development.

The other reason I tend to lean toward try-and-fix is that it's just not possible to plan perfectly. No matter how many planning meetings you have, how much documentation you write up in advance, how many tests you prepare, something is going to go wrong and have to be fixed. So no matter how much you plan, you still need the flexibility to respond to issues that come up. If you have that flexibility, why not use it as much as possible?

[Anonymous Monk]

It's a lot easier and cheaper to tear down a program and start over than it is to tear down a house and start over.

That is why all the real geniuses design their house on paper, then build a paper house (or bamboo and tarp) and live in it for a while (or do some walkthroughs)

[zentara]

The common thread in each of these, it seems to me, is that “we are being paid to Write Code, therefore let us Write Code constantly.” Let’s go from Something We Can Release to Something Else We Can Release every few hours.

I think you are witnessing the current Tech Bubble, wherein all the overpaid, over-hyped, under-worked programmers are desparately trying to justify their jobs.

It also has some connection to Wall Street, where traders are desparate to find an industry wherein they can blow big bubbles.

Did you get zucked by Facebook's IPO? :-)

[talexb]

I used the agile methodology at last workplace, and I use it now at my current workplace. And both approaches were different.

At the old place, we'd gather at about 10am and have a quick stand-up meeting: 1. Here's what I did yesterday; 2. Here's what I plan to do today; and 3. This is what's blocking me. Everyone got 1 minute; committee business would get put off until later. And it worked fairly well.

At the new place, every work item is costed out very carefully, we update our issues regularly, and we meet daily. It sounds roughly the same, but it feels like *much* more process, and a bit cumbersome.

The waterfall approach is close to what I learned at university (Systems Design Engineering at Waterloo), but it's a more academic approach than what I think really works well in the real world. In the real world, you don't want to spend weeks writing a perfectly detailed specification on what the system's going to do. It's much more productive to have a good general idea of where you're starting from, and be able to show a version that does a little bit, but does it well, shortly after starting. That's a much better approach than coding for eight months, finally having a demo of the system and hearing "Nah, that's not right."

Building software is *not* like building a house or a bridge. You don't think about rebuilding a house's entire foundation after it's been in place for 50 years, but, if the software's structured correctly, that type of thing is, or should be, fairly straightforward for a legacy system.

Really, agile should be about finding a good balance between planning and doing. That's all.

[sundialsvc4]

Bouncing a bit off of your fourth paragraph, let me play Devil’s advocate for a sec.   I wonder if it could be argued that ... if you spent X months building something to be told then that “that’s not right,” then you obviously didn’t build your blueprints.   I wonder also if it could be argued that what you are suggesting is actually an apologetic for, “let’s just make the whole thing up little-by-little as we go along.”  

“If you do not know where you are going, you will get there ...”

If one oh-so confidently asserts that software should be designed on-the-fly whereas a building or a bridge (that maybe costs less) requires rigorous design and specification ... what is it, exactly, about software that justifies exempting it from the advance planning process that is required even of a company that fixes pot-holes in the street?   (The original builders are frequently held liable for those pot-holes, by the way...)

Are you seriously telling me that the only way to determine whether a system will be capable of supporting the intended transaction volume is to build the thing, throw it into the water, and see (for the first time, as it were...) if it swims?!

I am unconvinced that “there is only one way to find out...”   You can’t figure out where to put in a septic tank, much less actually put one in, without doing a soil-percolation test first.   (Phew.)   When the plans are drawn-up and approved, this sort of preliminary investigation and calculation has already been completed.

This Devil’s advocate says ... “Baloney!”   What say ye now?

[chromatic]

What say ye now?

I say 40 years of people misunderstanding Dr. Royce, failing, and saying "It would have worked if we'd only BDUF more!" and failing just as badly next time gives me little confidence that BDUF more next time will ever work.

[talexb]

To follow your boat analogy, I'd build the hull and make sure that it floats, draws the depth that I expected, performs in the wave tank correctly, and then continue with the deck, the fittings, the rigging, engine/sails, nav equipment and so forth.

I may not have explained my 'on the fly' approach very well: I'm talking about getting started with the basics, and making sure that the basics work correctly.

[Jenda]

The problem is that the building analogy doesn't hold. Yes, you have a sketch, then the approximate design, then more detailed design, you add more details here, change some details there ... the catch is that when it comes to software, the detailed design is the thing you are after. There is no "OK, now we have the plan, let's order the bricks and build the thing. In twenty copies." The detailed plan is the final product.

[zwon]

What do they do? They plan. They plan absolutely everything, and they do it well in advance.

Uhm... I guess you've never been involved in building Big Heavy Things ;) They sometimes (quite often actually) miss targets by years and billions

[sundialsvc4]

Every human endeavor is marked by Kilroy.   But that does not obviate the planning and execution process.   How much more drastic would these failures be if the construction company had been making it up by the seat of their pants.

[zwon]

But that does not obviate the planning and execution process.

Isn't agile all about planning and execution? The problem with Waterfall is that their long-term plans are not reliable.

[tobyink]

There does seem to be one major difference between software and houses that you've not taken into account.

If you get into enough detail planning software, you arrive at a point where you're writing algorithms, and you realise that you're not just planning the software, you're coding it.

But no matter how detailed architectural drawings get, you can't live in them.

[sundialsvc4]

I think that this is precisely the thing that silver-bullets like Agile et al apologize for:   “you realize that you’re building the house on-the-fly, just like you’ve always done, therefore, just keep doing it.   After all, you wind up with a visible structure that much sooner ...

After all, software development is so different from anything else that we have done throughout all of human history, “it must play by different rules.”

After all, users really won’t mind if the project is “a little late,” because they can plainly see that we are hammering nails and cutting boards as fast as we are able.

(thin smile...)   If only it were actually “a little.”

I have consistently found that, if you plan the project as thoroughly and as meticulously as you actually need to, the actual construction of the thing is, as I said, “virtually an afterthought.” It also enables you to say, and not only to say but to say consistently:   “deployed on time, deployed on budget, zero defects.”

z - e - r - o .

You know, there is really nothing complicated about writing, say, “Perl code that works.”   Writing the stuff is actually not complicated at all.   Building a test suite is also not very complicated if you are building it from a pre-arranged laundry list.   No, I submit for the consideration of the Monks that the reason why software projects so consistently overrun, in every possible way and at sometimes awful cost, is that we are making the stuff up as we go along, and we are making excuses for shoddy (non-)engineering practices that would never pass muster in any other engineering discipline.   Computer software is punishingly intolerant of this, because it is composed of thousands of moving parts.   And yet, we do it, and we insist that we must.

[chromatic]

If you redefine "programming" to be "the mere act of typing", then it's easy to "program" on time and under budget.

If you always have customers who know exactly what they want before you type a line of code and they never change their minds or ask you for anything else and they're happy when you never let them, you're the first programmer I've ever talked to with those characteristics.

Change happens.

[aaron_baugher]

I have consistently found that, if you plan the project as thoroughly and as meticulously as you actually need to, the actual construction of the thing is, as I said, “virtually an afterthought.” It also enables you to say, and not only to say but to say consistently: “deployed on time, deployed on budget, zero defects.”

I have consistently found that the above paragraph is nonsense.

Now, when a task is small enough that I can plan the entire thing out in my head before I write a line of code, then yes, I can predict pretty accurately how long it will take and cost. But that's because, as someone said elsewhere in this thread, "planning" in the context of programming IS coding to a significant extent. If you're planning and thinking about the program in terms of code and algorithms, then once you have it all "planned," you already have it coded, except for the tedious typing. Of course you can provide solid deadlines and cost estimates then -- the real work is done.

But that just redefines programming as planning, in my opinion. How about if you're writing pseudo-code? Is that planning or programming? What's the difference? What if, instead of writing pseudo-code and flowcharts, I go ahead and start writing real code, tweaking it as my "plan" evolves? How do I tell when I'm planning and when I'm coding? Maybe you can plan a program fully without thinking through the required code (and database tables and so on), but I can't. I don't even know what that would mean.

Look at it this way: instead of comparing building a program to building a house in real life, compare it to building a house in a CAD package (apples to apples). You could plan the house out on paper, with lots of notes and hand drawings about what you want until you have complete blueprints, and then use the tools in the CAD package to "build" the actual 3D house, considering that a separate step from the planning. Or you could just start up your CAD package, load a basic template that has some algorithms you commonly use (walls, floors, roof), and then start adjusting and tweaking until you have the house you want. Odds are you'd do a combination of the two: maybe a couple quick sketches on paper to get the basic look in your head, and then start "building," working out most of the details during the building stage.

You condescendingly imply that we're being ridiculous when we say programming is different from "other engineering disciplines," but the fact is that programming is NOT engineering, at least not in the full sense of the discipline. Building ideas from smaller ideas is different from building objects from smaller objects, like it or not. It presents different pros and cons, and the fact that "planning" and "building" can be intermingled in a way that they can't be in brick-and-mortar projects is both pro and con. You can't get away from that con, so you might as well take advantage of the pro.

[DrHyde]

The reason that we can know in advance exactly how much wire we'll need when building a house is because houses are very simple systems, and because we have literally hundreds of years of institutional experience of building things like houses.

But even then, there is waste because they never seem to order exactly the right length of wire, exactly the right number of bolts, exactly the right amount of sand and so on, so even though in theory we can know this, in practice we don't because it is cheaper to make an estimate and get the product finished quickly than to spend ages figuring it all out in advance.

My experience with "Agile Systems" has been that, for small tasks, estimates and actual delivery do tend to converge over a reasonably short period of time. For large tasks (those which get broken down into smaller tasks, and those into still smaller tasks, and so on), they tend to be accurate to within an order of magnitude - that is, you can confidently tell the customer that it'll be done in weeks, but not in months, or in months but not years. Funnily enough, that's not that far off the sort of accuracy of estimates that you see in an awful lot of large "building heavy things" engineering. It's common for big engineering projects to come in late or over budget, but not for them to be an order of magnitude late or over budget.

The real benefit of agile working practices, though, is that constant feedback, constant estimating, involvement of everyone in those, and a culture of "how do we fix this" instead of "who do we blame" lets you catch overruns early. Catch them early and you can either fix the problem and get back on track or you can "manage your customer's expectations". Customers generally don't mind things being a bit late, as long as they know in advance and so can plan for it. And even if they do mind, it's still better to tell them, because they'll respect your honesty.

[doom]

There's clearly a range of jobs where you need to use some form of "waterfall" (better, I think, would be to call it "Design", or more precisely, "detailed, upfront design"). An obvious example is something like the control system for a nuclear power plant.

As a number of people have pointed out, "design-then-build" doesn't actually describe the construction industry terribly well, and myself I'd suggest that everywhere and anywhere you look, there's a tension between upfront planning and reacting to circumstances. One would not, for example (brace yourself: car metaphor coming) try to drive a car by calculating the angle to turn the steering wheel and then holding it there, irrespective of where the car was actually going.

There's a role for iterative, evolutionary approaches.

[sundialsvc4]

Important:   Let every Monk please understand that, although I am asserting my two-cents strongly and doing so sometimes in reply to someone else’s prior comments, I mean anything and everything that I say as my opinion only, and not personally.   I am standing on one soapbox and you may be standing on another, but you are never, ever, either the subject nor the target of anything I might say or have said or might say.