Re: Nobody Expects the Agile Imposition (Part VI): Architecture

[ELISHEVA]

I've been hesitant to respond because I know you've put a lot of work and thought into this essay, but there is something here that just isn't ringing true for me. I'm struggling to put it into words, but I think it might be this.

The entire essay relies very heavily on three assumptions:

• There is a clearly recognizable distinction between refactoring and rewriting that is so obvious that it doesn't need to be explained. Rewriting is bad and loses information. Refactoring is good, but only if done continually and with the help of tests.
• Satisfying the customer is good and over design is bad. The difference between "design what the customer needs and no more" and over design is clear and also doesn't need explaining.
• Human beings, and particularly crack programmers, are largely incentivized by external rewards - customer statisfaction, money, BMW, etc.

The problem with such undefined and assumed distinctions is that they are hard to apply to any sort of real world situation or even to discuss and debate.

Rewriting vs. Refactoring

I think once you start trying to define what one really means by "rewriting" vs. "refactoring" one quickly begins to realize that neither "rewriting" nor "refactoring" is really the issue here. Suppose I start with a clean empty file - am I rewriting or refactoring? Now suppose I fill that file and three others carefully, one subroutine at a time as I identify function points and data in the original code and apportion them between the three class files. Now, am I refactoring or rewriting? Now suppose the decision to split the former single file into three is based on a sub-system model developed on the basis of 3 months research into alternative models for that subsystem, the history of failed and successful standards for that problem domain, plus some original research work. Now am I merely refactoring or rewriting?

I would hazard to guess that the well-researched, well-investigated and well-thought out rewritten module is likely to last longer and require less maintenance than the code that was merely refactored to meet the latest client need using no more input than old code and current needs. If that is the case, then the determining factor is not rewriting vs. refactoring, but rather "well-thought out and researched" vs. "consider only what is before my eyes".

What is over-design?

Is over-design anything beyond extracting and studying customer stories? Is over-design researching standards? Is over-design understanding the business theory and context of a customer problem? Is over-design exploring alternate architectures and assessing them in terms of growth potential? When is research too much and when is it just enough?

The classic answer is "over-design" is going beyond the customer needs or trying to nail down details in your head that are best nailed down by doing. But this only brings us to the next question.

What exactly do we mean by "customer needs"?

Can customer needs really be assessed without research and knowledge apart from the customer? How is acquiring that knowledge different from over-design? In my experience those needs can look very very different depending on the questions you ask and the background knowledge you bring to those questions.

Suppose I go to customer who has only used cash-basis accounting and say "what do you need?" Since they've only ever used cash basis accounting, they describe to me a cash system.

Now if I know nothing about accounting, I'm liable to take that at face value and build/buy them a cash accounting system. However, if I know something about accounting theory and typical growth patterns of small businesses I'm likely to probe further. Most expanding businesses (not all) outgrow cash basis accounting rather quickly and I'd rather get a read on their near term needs. In this case, after a brief conversation it becomes clear that the potential customer should have been doing accrual accounting long ago. They just had a conversation about this with their accountant and they are simply afraid to make the switch. If I give them a cash system, it needs to have a migration path to accrual basis. If I give them an accrual system, it needs to have a strong training program. This is a very different set of requirements from my first "tell me what you want and I'll do just that" set of questions.

I've seen this "know the right questions" story many many times. Until you ask, people aren't always aware of cross department information flows that the new system needs to satisfy. Until you ask, people often forget that some larger customers have multiple accounts from a sales point of view but a single account from a legal/billing point of view. Until you ask, people aren't always aware that with a small tweak in their way of viewing information, a whole new product or service line could open up. Though this isn't something they want to do now, if there is a way to build the system just as quickly from the start and give the ability to plug in new modules to support these services, they would very much like that. Further probing shows that product lines come and go very quickly so that if not these products, then some other batch of product opportunities is likely to develop and quickly obsolete a fixed system. (Gasp - they have a business need for a framework architecture?).

What really motivates people?

Are all people driven by this velocity you talk about? By the opportunity to have a BMW in the right spot in the parking lot? Some are, but others aren't. If not, how does that affect the way you manage a project? Will focusing so much on velocity promote incentive or unintentionally kill it? I think it depends very much on the team. There is no formula and no way around tuning management practices to the individuals involved in the work. That is what makes good management hard work.

When I was doing my MSc. in management one of my professors had us take a test on our own work motivations. Then he presented studies carried out over a much wider range of the work force. The point he drove home was that the people in our classroom (all upper-mid and senior managers) had very different motivations than the larger workforce.

During the height of the dot.com boom, a friend of mine did a compensation study for a lagging start-up. The founder wanted to know what he could do to get better performance from his team. He'd tried options. He'd tried raises. It didn't work. What did they want? Toys. Cool graphics equipment that was a pleasure to use and write code on. They spent so many hours at work that what they wanted most was for the work environment to be a fun place to be.

What is success?

If assessing customer needs requires at least some background research, then perhaps we need a deeper rule for sorting out what counts as good research/design vs. over-design. Perhaps success should be the measure? If so, what measure of success do we use?

Let's take an extreme but concrete example. Is *nix a success story or a failure? By one measure it is a total failure. Most of the companies involved in supporting it have either gone out of business or been absorbed by larger entities (most recently Sun eaten by Oracle). By another standard it is a crazy success story.

By common definitions of design, Unix and its *nix cousins could possibly be one of the most overdesigned systems in computing history. It had a Gig of addressable memory when 1Meg of RAM was considered huge and the largest chips came nowhere close to a Gig. It identified every system resource with a path even when they looked nothing alike to the end user. There was no market research, story boards, or end-user discussions. Its design was driven by deep, even academic, reflection and questioning on the nature of operating systems, how operating systems could and should work.

Microsoft DOS on the other hand is a good example of design what you need. According to a book written in the 90's on the history of the operating system (title is buried in storage in the US so can't cite), the original DOS operating system was stuck with 16 bit pointers because it was designed for a chip that had only that much addressable space. Why waste computing power on a pointer that was longer than the addressable space of any known memory chip at the time?

Because of its design-to-need decisions Microsoft has spent literally billions of dollars first patching and then discarding operating systems. It has spent billions more on marketing explaining away product delays, bloated resource requirements, breakage of software that depended on older operating system features and assumptions, security breaches, and so on. Thankfully, for Microsoft, it has the money for all that spin, but is this success?

Obviously, the Microsoft family of operating systems have been a commercial success. However, there is more than one kind of success in the world. Remember that poem about Ozymandias:

And on the pedestal these words appear: "My name is Ozymandias, king of kings: Look on my works, ye Mighty, and despair!" Nothing beside remains. Round the decay Of that colossal wreck, boundless and bare The lone and level sands stretch far away

Survivorship is another form of success. I wager to guess, given the huge and increasing resource requirements of each successive generation of Microsoft OS (we're now up to a minimum of 2Gig's recommended for Win7), Microsoft's OS will eventually end up in the history books. If the company survives, it will be by virtue of re-invention, along the lines of IBM who saved itself by leveraging strong corporate relationships and sacrificing an 80+ year history of selling business hardware to turn itself into a consulting services company at the end of the 20th century. making the transition from type-writers to computers and business IT design in the late 80's and 90's

The *nixes on the other hand, despite their lack-luster commercial success seems to have staying power. I would not at all be surprised if it 100 years from now the next generation of operating systems traces its ancestry to some form of *nix. It is often the system of choice for the increasing array of smart personal accessories and home equipment.

This is partly due to it being open source and partly due to its small footprint and stability. Code bases come and go, but the core architecture hasn't changed in literally decades. Truth be told, a large part of NT is actually borrowing from *nix - one reason why Microsoft (FUD aside) was so keen to own and protect that IP. (can't find supporting citations)

Final thoughts

In my own work, I've always been aggressively focused on customer needs, but I've always also coupled it with intense research. I don't think one can exist without the other. The research helps me (and my team) put the needs in context and avoid short-sighted decisions. The customer needs keep the research focused on a particular problem rather than interesting tangents. I'm uncomfortable with the sense of either/or that seems to pervade many discussions of scrum/agile. The technique seems to be becoming more important than the cooperative judgment calls it was meant to facilitate.

Gradual refactoring is a technique I often use (refactor a little, test a little), but it only works when you have (a) a clear vision of the end goal and (b) that end goal is a better architecture than you started with rather than one with just different mistakes.

I think 'scrum' has its place, but not as a model for all project management every where and anywhere. It works very well when either (a) some critical mass of people involved (either the user or the team) has broad familiarity with the problem domain and can integrate specific needs and background knowledge (b) no one really knows the domain, but the team members are determined creative learners who without even thinking much about it sweep wide for knowledge and reflect deeply on the problem at hand.

When the above criteria are met, then scrum enables such workers to thrive. When the above criteria are not met, scrum/agile design can quickly turn into a case of the blind leading the blind. You'll still end up with a Jenda tower even if you refactor a little bit everyday.

Update: struck out disputed line for which I can't find supporting citations.

Update: reworded the description of IBM's reinvention. After reflection the wording was confusing.

[roboticus]

ELISHEVA:

Rewriting vs. Refactoring

If you start from a blank source file, I'd consider that rewriting, even if you start transplanting subroutines from the original back into the new file. I feel that taking a working system and transforming it without breaking it is refactoring, while building a new thing (even when borrowing heavily) from the original is rewriting. Of course, you can do both in the same project, as you may refactor some bits and rewrite other bits.

The problem I generally see with rewriting is that there's quite a bit of knowledge that's encoded into the system that's not immediately obvious. Things like:

• Order dependencies: Systems consuming the output expect a particular sequence of records or operations and the order isn't documented.
• Workarounds for problems in other systems: Sometimes bugs become part of the standard interface, and the documentation isn't updated. So writing fresh from the spec causes you to rediscover those issues.
Refactoring addresses these (somewhat) by making a change at a time. By writing tests for all the changes you'll hopefully capture some of this hidden knowledge in your tests. They won't necessarily be documented any better, but when you put it into production and other systems break, you should be able to trace back to the original test and document accordingly.

Of course, refactoring has its own issues. If you can imagine a cleaner structure for your program and try to refactor towards it, you'll find that "you can't get there from here", or you have to first go to Timbuktu before you can get back home.

With my experience (quite a bit), I find that the less you know about the domain or the application, the more you should lean toward refactoring. Similarly, the closer you are to being a domain expert, the more sense rewriting can make. The problem is that it's often difficult to objectively judge just how much knowledge you have about the domain. My approach is normally to find some dividing lines in the system where I can break it apart with the fewest changes possible. Then I can choose to refactor some chunks and rewrite others.

Overdesign and customer needs

You've pretty much hit the nail on the head: If I'm paraphrasing your arguments correctly, the difference between overengineering or not is largely communication. If you're going off and doing anything beyond what you've discussed with the customer, you're overengineering things. If you think the system needs to do something specific, or that the architecture needs to go in a certain direction, you need to have a talk with the customer about anticipated future changes so you can shape things correctly. So if you discuss things with the customer and get buy-in, then you're doing your job correctly. If you have discussed things with the customer, and they're adamant about a particular direction, then you need to do what they want, or you're throwing their money away.

(Gasp - they have a business need for a framework architecture?)

I've heard it mentioned that "functionality is an asset, while code is a liability". Too often, programmers know they need some functionality, but build their own rather than buying it. (I, unfortunately, succumb to this temptation a bit too often, myself.) I've been trying to periodically take a break from design and/or coding so I can sit back and review the requirements so I can stop myself from going off into the rabbit holes.

...roboticus

When your only tool is a hammer, all problems look like your thumb.

[sundialsvc4]

To be quite frank, I believe that the essential difference between “refactoring” and “rewriting” is that one term is politically expedient, while the other term is not.   In both cases, you are doing the exact same thing in terms of the code:   you’re replacing the existing code with something altogether new, which renders the code inoperable (un-compileable) for an extended period of time and which must, in the end, be re-validated to verify that the new code works the same as the old.   The term, “refactoring,” is currently sexy and implies improvement ... “making an already-good thing better” ... whereas “rewriting” (wrongly...) implies previous failure.

I do admit to the reality that, sometimes, in order to get approval and funding to do what badly needs to be done, you are obliged to resort to “necessary euphemisms.”

Like it or not, computer software is very fragile (and therefore, costly) stuff, simply because it is riddled with functional and data dependencies.   It is, so to speak, “a house of cards,” which can only stand up to a very limited amount of “remodeling.”   I simply think that this is ... the essential and unavoidable nature of the beast.   It obligates us to try to do the best that we can, knowing that there are serious limits to that.   I submit that there is no silver-bullet language or technique to be had.   (He would rightly be a gadzillionaire who discovered it.)

With regard to the point of “overdesign and customer needs,” there is the consideration that (a) the customer does not always know just where his business will take him; and (b) in any case, he is not a professional software builder and does not profess to be.   Sometimes you do need to “go beyond what you discussed with the customer,” because in your professional judgment as a software engineer, those additional elements (for example...) create the foundations for future characteristics of the system that are reasonably foreseeable as well as engineering-practical.   But, you need to be sure that you get all points about what the customer requests, and of what you have in turn decided to do, and every single subsequent change to the foregoing, in writing and signed-off and filed away for all eternity.

Part of the (successful) argument for “frameworks” is that the cost of developing and maintaining them can be cost-amortized (or simply “unpaid-effort amortized”) among many projects that employ them ... thus allowing all of those projects to enjoy the full benefits without incurring the full costs.   The use of frameworks imposes a certain specific “world view” upon the project, however ... namely, the world-view of that particular framework’s designers, quirks and oddities and all.   Choose your project’s spouse very carefully.   The project’s entire future direction is necessarily molded around that of the framework, and in a very rigid way, except to the extent that the project’s actual implementation might be, by deliberate choice, architecturally divided into (framework-based) “client” and (non framework-based) “server” portions.   The cost/benefit analysis of using frameworks usually prevails in spite of this consideration, because so much of the constituent code in so many projects isn’t unique at all.

[roboticus]

In both cases, you are doing the exact same thing in terms of the code: you’re replacing the existing code with something altogether new, which renders the code inoperable (un-compileable) for an extended period of time and which must, in the end, be re-validated to verify that the new code works the same as the old.

In my prior message, I mentioned that refactoring is a transformation without breaking the system. BrowserUk mentions later in this thread that refactoring is an iterative process, and he's spot on. For example, if I inherit a chunk of code with a mess of global variables and procedural functions and I want to transform it into an object-oriented system, then I'll do it stepwise with the system working after each step.

First, I may decide to group related variables together into their own namespaces. So I'll create a new module for the new namespace, and put the variables in it. Then I'll update all the references to the variable to the new namespace. Then compile and test it. Then I'll move the functions appropriate to that namespace into the new modules, compile and test. After each individual step, the system still runs.

It's true that the system needs to be revalidated after you're done. But you should maintain the system in working order throughout the process. If revalidation fails, then it indicates a failure of your test suite/data to cover all cases. So be sure to update your tests, too! Since you're doing it one step at a time, it's pretty easy to cruise through many iterations in a day. No iteration is drastically different than the one before it, but if you know where you're going, then your initial state and end state will be dramatically different.

I could go on, but better authors than me have covered it well:

• http://c2.com/cgi/wiki?WhatIsRefactoring
• http://www.refactoring.com/
• http://martinfowler.com/distributedComputing/refactoring.pdf
• http://sourcemaking.com/refactoring

...roboticus

When your only tool is a hammer, all problems look like your thumb.

[Anonymous Monk]

rewrite simply means write again, it doesn't imply any improvement, where as refactoring implies improvements in readability/maintainability/estensibility/test suite

[eyepopslikeamosquito]

Are all people driven by this velocity you talk about? By the opportunity to have a BMW in the right spot in the parking lot? Some are, but others aren't. If not, how does that affect the way you manage a project? Will focusing so much on velocity promote incentive or unintentionally kill it? I think it depends very much on the team. There is no formula and no way around tuning management practices to the individuals involved in the work. That is what makes good management hard work.

*applause*. I couldn't agree more. I get intensely irritated when I see people being rewarded for writing unclean code at high velocity, being promoted to a new job, while I'm left to clean up their mess. If I can find the motivation (noone is offering me a BMW;-), I'll discuss intrinsic versus extrinsic motivators and other management issues at length in a future installment of this series.

[BrowserUk]

IBM who saved itself by making the transition from type-writers to computers and business IT design in the late 80's and 90's

Oh dear!

Truth be told, a large part of NT is actually borrowing from *nix

Oh dear, oh dear.

If you are going to write authoritatively about history, it would really be better if you actually knew something about it.

[Corion]

Actually, IBM restructured itself (and thereby I guess saved itself) by moving from a hardware+OS vendor (OS360) to a consulting company (also see its purchase of PWC), so that part isn't that far fetched.

[BrowserUk]

I was working for them when the transition occurred. But it certainly wasn't from typewriters to computers.

[ELISHEVA]

The comment about NT and *nix - that was based on my memory of press reports at the time it was being developed. If I recall correctly they originally wanted to do a green field system and then found that they had to borrow certain parts of the *nix architecture - what exactly I don't remember. I know many of the developers came from DEC, but the few things I'm finding on the web focus on the VMS influence. Business press reports on technology often get it wrong, so I might be remembering someone reporting the DEC hirings and just assuming it was DEC UNIX rather than VMS that ended up in NT.

[eyepopslikeamosquito]

I also got the early history of NT wrong going from memory. That won't stop me from continuing though and I'm sure BrowserUk will correct me if I'm wrong again. :-) I tried doing some googling just now but couldn't find any citations to support a couple of anecdotes I remember reading about years ago. I would have read these in a book not the press, but can't remember which one. One was that Dave Cutler was a very "passionate" guy, so much so that on one occasion he punched a hole in the wall when one of his junior programmers disobeyed him and made a "safe" last minute change that broke an OS release. Apparently, the hole he punched in the wall was later cut out, framed, and mounted on the wall of Dave's office.

I also remember reading he was pretty passionate about not liking Unix very much, one quote I remember was that Win32 will never have Unix-like signals because Dave shouted "Signals are a crock!". BTW, I'm not a fan of signals either - especially when writing (async-signal-safe) signal handlers - and signals certainly don't mix very well with threads. Update: see also Re: Signals vs. Windows by BrowserUk

Another anecdote I remember was Dave responding angrily when one of his team said "Unix did it this way". Dave responded something like "Just because Unix did it this way, it doesn't mean it's right". Anyway, the overall impression I got was that Windows NT was strongly influenced by the (non-Unix) Digital operating systems Dave and the other Digital guys hired by Gates had worked on.

Apart from the history and the anecdotes, the Win32 API "feels" very different to Unix - for example, compare and contrast the many complex parameters and sub parameters of the Win32 CreateProcess call with Unix fork and exec. Generally, Unix system calls have far fewer parameters than Win32 ones.

[eyepopslikeamosquito]

I think 'scrum' has its place, but not as a model for all project management every where and anywhere.

I agree. I hope this whole series of articles (especially the first one) has made that clear. If it's appropriate and the team wants to use it, knock yourself out with Scrum, but do not impose it on the team from outside. For the record, while I generally support agile and lean principles, I prefer to think for myself rather than blindly follow a "branded" methodology. If forced to choose a "branded" methodology, I'd choose Kanban.

[eyepopslikeamosquito]

This is partly due to it being open source and partly due to its small footprint and stability. Code bases come and go, but the core architecture hasn't changed in literally decades.

I guess that depends how you define "core architecture". :) I'd say there are at least three competing core architectures for implementing Unix:
• Monolithic kernel
• Microkernel
• Hybrid kernel
The infamous 1992 Linux is obsolete debate between old hand respected operating system researcher Andrew Tanenbaum and young upstart Linus Torvalds makes interesting reading. I guess it shows that "theoretical (academic) superiority" does not necessarily translate to success in the marketplace.

I think it's fair to say that monolithic kernels still dominate the Unix arena, though Tru64 UNIX is built on top of the Mach microkernel and Mac OS X is built on the XNU Mach/BSD-based hybrid kernel. Though I'd like to see the microkernel based GNU Hurd succeed, sadly that now looks doubtful after more than twenty years of development -- yet another example of the perils associated with "writing new systems from scratch".

[tilly]

It is worth noting that I personally know the original GNU architect, and he claims that he thought at the time that the easiest way forward was to build on top of BSD. But RMS chose Mach, in part because academia was very fond of microkernels at that point.

He says that RMS has acknowledged that this decision was a mistake.

[eyepopslikeamosquito]

The difference between "design what the customer needs and no more" and over design is clear and also doesn't need explaining.

To clarify, I'm not an unthinking follower of "design what the customer needs and no more"; I feel that's a dangerous over-simplification. While writing code that is never used is certainly waste, and one has to beware of over-engineering, I don't view this as a black and white issue. I touched on this in the "Agile Design" section where I stated:

Software design is an art requiring experience, talent, good taste, and deep domain, computer science and software usability knowledge. I feel there's a bit more to it than the four simple rules above.

and then continued on to present my twenty tortuous rules. :)

[sundialsvc4]

This post is proof-positive that, when you see More... at the bottom of what seems to be a very short posting, it pays to click on it.   Too bad I can only vote it up “once.”