Or, How I Lost My Battle With The Blob

This is a cautionary tale about implementing refactoring in the workplace. I have always been in favor of refactoring bad code, and I think getting others to work with you in doing this is a great idea. However, if you don't exercise a little caution, it can become a major disaster.

I looked up from my screen one day to find my project manager leaning over me. "Stephen," he said, "I need your help." It seemed that management had promised several clients that we would add a huge number of features to The Blob by the end of the month.

Now, "The Blob" is what I'll call the company's main project. It was a sprawling application that had originally been written in Perl 4. It was originally supposed to be a quick one-off application, but it had been growing steadily for the past two years, feeding on a rich diet of quick customer feature requests. Inside, it was a tangled, seething mass of global variables, if-then chains, and copy-and-paste code. Most of the programmers who had ever worked on it refused to ever face it again. A huge effort was underway to rewrite The Blob from scratch, but it had hit significant delays, since the Blob kept growing and changing shape.

It sounded like fun.

See, I had been studying more and more about refactoring. I was pulling together my collection of Perl refactorings, and I figured that this project was a golden opportunity to learn more. It would be absolutely impossible to add the features the customer wanted without some severe refactoring going on. Management wanted to assign me to this project for a week. I figured that that meant three weeks, at which time I'd be back on my own project again. And hey, they needed my help.

There were two programmers who had been working on the code already. Both were very bright. Neither of them knew anything about refactoring, and were very much of the "slap a patch and run" style of coding. I decided on a two-pronged approach. On one level, I would teach them as much as I could about refactoring, automated unit tests, CVS, and all the other tips and tricks I'd picked up. On the second level, I would practice what I preached. I would write unit tests for any code that I touched. I would refactor the complex code I encountered into orthogonal, easy-to-comprehend code that we could then extend more easily. Easy, I figured. I learn by teaching, these guys pick up some valuable skills, the code gets improved, the changes get made. Everybody wins.

I'd been working alone WAY too long. Down in the murky depths, The Blob was chuckling softly to itself.

At first, everything went extremely well. The other coders thought refactoring was a great idea. One of them spent a day reformatting a huge module to suit how he liked his code; I hadn't the heart to explain that that wasn't what I meant by refactoring. Still, the intent was good, and I sent out a steady stream of e-mails explaining what I felt refactoring was. In the meantime, I was pulling apart gooey piles of Blob-flesh into neat, object-oriented modules. I wrote compatibility interfaces to keep the unrefactored code happy, built my automated unit tests, and was pleased to see them run.

The reversal began slowly, almost imperceptibly. The other coders proclaimed my style as an excellent one-- but never adopted it for their own. Automated unit tests were a great idea-- but no one but me ever wrote them. Even more frighteningly, no one but me ever ran them. No one had the time, it seemed. Then my refactoring started to take the blame for things. Despite the fact that I had automated unit tests backing me up, somehow I was always the one called in to fix emergency problems-- and since I had fixed the problems, somehow it was assumed I had caused them.

Blame culture started seeping in everywhere. I would spend time refactoring an area-- then discover that another programmer had reversed the changes. The two other coders and I, normally calm and reasonable people, started sniping behind each others' backs. Sick of hearing false alarms about errors supposedly caused by my code, I became increasingly defensive, and found myself blaming people right back. Communication between the programmers became more and more difficult. The Blob, far from being defeated, was now in our very company culture-- where, in fact, it had been from the start.

It took a long time for me to realize that, although the other coders were intelligent people, they saw the world very differently than I did. I found the pell-mell use of globals disturbing and dangerous, and refactored them into constants and method calls. They were used to the global variables, had never used references, still used 'local'. They found my constants and method calls just as scary as I found their global variables. They weren't used to comments, so they didn't read mine. They agreed with my ideas, but I was asking too much.

I was asking them to adopt and understand my style. And we'd all been working alone for far too long.

The Blob lives on to this day. No one has yet tamed it. Now, long after the blob-taming project has ground to an ambiguous halt, I have some ideas on what I should have done that might have done a better job at Blobicide. Here they are:

Devise A Common Goal

In the beginning, I thought that the other programmers shared my goal-- a well-factored, object-oriented system, easy to expand and debug. However, the other programmers didn't see it that way. They liked my ideas, but they'd been working with this code for months and were used to it. Warnings which set off screaming alarms in my head were peace and quiet to them. My code, on the other hand, was not comprehensible to them. Much as I prided myself on my easy-to-read style and verbose commenting, it might as well have been an obfu to them.

If we had, from the outset, agreed on a certain style, it could have worked. It's easy to think, after one has been working alone for months upon months, that one's coding style is the ultimate one. After all, I understand it as easy as reading; why can't they? (Because they didn't write it, that's why.)

Don't expect your coworkers to change their style if you aren't willing to change yours.

Who's The Boss

In trying to radically change the practices of this little coding team, I was essentially trying to take it over without admitting it. Small coding groups don't always need a clearly defined leader, but subtle power struggles can easily lead to escalations in blame culture. If I had shared my goal of a well-factored system in the beginning, and asked for them to follow me-- and been willing to follow their lead if they weren't willing to follow mine-- perhaps things would have ended differently. We might not have tamed the Blob. But we would have had a goal we all were working toward.

We were hampered by the fact that our official project manager was so hamstrung by management that he was, essentially, reduced to the role of go-between. Management buy-in would've been nice, but management couldn't understand a word of the entire debate that was going on.

Don't Force Knowledge

For me, this is the hardest one to learn. I always like to teach people things that I've learned. However, I've gradually come to realize that no one-- programmers especially-- will learn things when you tell them actively. However, if you indicate that you have this knowledge, give a little demo, then wait for them to ask for more, then you'll never have better students. It's risky. They may never ask. They may learn on their own. They may get something wrong. But if they don't ask, they probably don't want to know.


Replies are listed 'Best First'.
Re: Failure To Refactor
by Rex(Wrecks) (Curate) on May 15, 2002 at 05:39 UTC
    ++ on the node, and sorry about the project.

    I worked on a similar project some years ago, and your ideas are very sound. I would add a couple that I have learned along the way as well. These have been crucial in the joint projects I have worked on. Your mileage may vary:

    Assign an Owner

    Projects need an Owner, and someone who is, ultimately, responsible for the outcome. Loose groups seldom work unless the group has worked together for some time and knows the others habits and styles. You touch on this in "Who's The Boss" above, but I would advocate that a "Leader" per-say may not be the same as an Owner. With an Owner you have acountability and therein lays one of the secrets. Someone is accountable!

    Get buyoff from Management

    Present a plan to management, and don't start until you get paper signed buyoff. This can be tricky, and details may change, but getting approval can save headaches later. Your plan should include who will be the Owner of the project and what resources and/or authority the Owner will require. This may not end up being you! But if it is you, present a clear goal to your team, and modify it with their input. Ideally this is done before the buyoff from management.

    I hope this gives some input and sheds some light for you. These are a couple things I use to keep my life sane when working on group projects.

    "Nothing is sure but death and taxes" I say combine the two and its death to all taxes!
      If you become the owner of one of these projects it is really important that you are free to choose who you have on your team. Ideally you want to keep the people who've been maintaining the code up to now, but if it becomes impossible to work with them then you must be able to either fire them (from the project) or be able to walk away yourself.

      Walking away is a drastic option, no question, but there may come a point where continuing to work with a team you don't get on with is doing you far more harm than good.

      Remember, if you're the Owner then you are responsible, do everything in your power to make sure that you deliver. And if you reach the point that you know you can't deliver, run, don't walk to your management and give them the bad news; putting it off will only make things worse. And if doing that loses you the job... well, if you can't change your organization, change your organization.

        "If you become the owner of one of these projects it is really important that you are free to choose who you have on your team."

        Update: Actually the above stament might be more true in smaller environments or startup situations. However I work for a large (50,000+ employees) company, and in my case it is very seldom that you get to pick and choose.

        In an Ideal World this may be true. Unfortunatly, in most cases you will be lucky to have input on who will be team members, let alone be able to choose. But you are right about one thing, you MUST have the authority to keep the group in line.

        And I also agree that management should be kept up to speed on progress, especially in the software world, where slipping dates has become the norm (and, no I don't like it anymore than anyone else).

        "Nothing is sure but death and taxes" I say combine the two and its death to all taxes!
Re: Failure To Refactor
by Zaxo (Archbishop) on May 15, 2002 at 01:10 UTC
Re: Failure To Refactor
by VSarkiss (Monsignor) on May 15, 2002 at 02:25 UTC

    Very nice, stephen; insightful and well-written, ++

    I've learned these lessons in over mumble years in the profession, and the sad part is that sometimes I still repeat the mistakes.

    Ultimately, you have to keep in mind that computer programming is a human activity, and people are people, whether they be programmers, managers, testers, or what-have-you. Their motivations may be different than yours, but equally valid nonetheless. Unless and until you learn to deal with your collaborators on a human level, your project won't succeed. Open-source types (for lack of a better term) who've been around a while end up learning this lesson in very public ways. ;-)

Re: Failure To Refactor
by perrin (Chancellor) on May 15, 2002 at 02:48 UTC
    Ouch, I've been there! Working with people who don't care about code quality is really rough. It's actually the main problem I see with these kind of XP practices: they only work if everyone does them. Sadly, Perl often ends up taking the blame for bad coding like this, even though it's ultimately the people who wrote the code who are at fault.
Re: Failure To Refactor
by P0w3rK!d (Pilgrim) on May 15, 2002 at 17:30 UTC

    How I Lost My Battle With The Blob

    It appears that you have captured everything I had to deal with in my last gig. I performed an extensive amount of design and development for ASP based projects and was pulled into some Oracle/Perl backend stuff. The refactoring came in 2 parts. The first being the rewrite of ASP code with embedded SQL and other hardcoded/static crap. The second being the refactoring of some backend shell scripts into Perl. The later turned out to be automating some projects in Perl by wrapping it around an existing Blob. My mistake was that I said "I know Perl" when asked by my development manager at the time.

    "The reversal began quickly. Management, developers, etc. proclaimed my style as an excellent one-- but only a few adopted it for their own. Automated unit tests were created for Perl. The ASP unit tests were manual. No one else performed any unit testing - less one developer. Even more frighteningly, no one but me ever ran them. No one had the time, it seemed. Then my refactoring started to take the blame for things. Despite the fact that I had automated unit tests backing me up, somehow I was always the one called in to fix emergency problems-- and since I had fixed the problems, somehow it was assumed I had caused them." (paraphrased from stephen) This was most obvious with the Oracle back-end team who f'd up every stored procedure they wrote. All their testing was either half-assed or nonexistent from what I could tell.

    Before I knew it I was working on multiple "Blobs". The backend team could not get their act together. Everytime something croaked (build process, infrastructure, oracle, etc.) is was my fault. False alarms occurred on an hourly basis.

    Devise a Common Goal

    That would be "Kill the front-end developer"...and if he writes Perl "Kill him and all his children".
    I can only be pulled in so many directions. It appears the development managers goal was to turn me into plastic wrap.

    Who's the Boss

    There was an obvious animosity between QA and development. I was beaten and mauled in every meeting by QA for bugs which were backend only. Regardless of the fact that my front-end code was complete and working. QA acted like Management and Management sat back and watched the fireworks. Management failed to do its job on many fronts.

    Common Goals

    The common goal in the development environment was not to cooperate. Coding standards which were non-existent when I arrived were created. 2 or 4 developers continued to do what they wanted. Management did nothing when it created problems. I guess the train was moving to fast for them and they were too busy worrying about their money.

    Don't Force Knowledge

    Never got the chance with all the other BS going on.

    Closing Arguments

    Refactoring is great when you have time, a professional environment of people (and I don't mean a bunch of suits), management with half-a-clue, and developers who know their stuff. Thanks for writing this up for me! :)