Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

what to do when you screw-up?

by arcnon (Monk)
on Feb 27, 2006 at 18:28 UTC ( #533103=perlmeditation: print w/ replies, xml ) Need Help??

I think everyone has bumped into a project that isn't going to take long.

In its startup phase you knock it out in 200 lines of code plus legacy code... This Legacy code was written before your employment and in a state that you deem barely useable.

Our program is simple and straightforward. We crank it out impressing the boss using the pre-built venerable code of your predecessors. You have the intention that you will fix all this legacy code in the future. And never in your wildest dreams believe that it will need new features on this little knock off application.

Low and behold 13100 lines later you realize you screwed the pooch. Many of the legacy modules that you included have reached their limit. The code we wrote has become difficult to maintain. Feature after feature has been added and has for all intents and purposes been rewritten 5 times... just never completely in one go.

---------------

I decided because of the amount of work that I have on my plate and it is MY fault that I'll work on a rewrite at home. But this 'little 1 shot' program now needs to be implemented 4 more times and I don't have enough time to 'do it right'. My boss who has never programmed a line just knows that the other departments who are requesting this app are being postponed by me and doesn't understand why I can't just let them use it 'as is'.

I can lay blame on aspects other than myself but in the end it all this falls to me… when you’re the 'working boss' that’s where the fault should lay after everything has being said. It still has to be fixed... it still has to be done right.

So do you just get it useable by everyone rewriting the old and dieing code? Allowing the separate apps to diverge at will... which will happen most likely happen reguardless anyway. Or make a strong argument for the rewrite and take the heat of it not being done yesterday? Is there other options that I should consider?

Comment on what to do when you screw-up?
Re: what to do when you screw-up?
by Tanktalus (Canon) on Feb 27, 2006 at 19:17 UTC

    First thing's first. You need to completely eradicate the line of thinking that says it's your fault that the legacy code isn't up to snuff.

    Next, you need to have a sit-down with your manager and have a little conversation about Technical Debt. And how bad it is now, and what it will take to fix it.

    This is a resource question. And managers get paid to make resource decisions. Don't deny your manager a full and honest reckoning of the cost of this application in real terms just because you have a mistaken sense of guilt.

    What you should have done, in an ideal world, is keep your manager in the loop of the potential problem as soon as you noticed it. However, now is not the time to dwell on such mistakes - it's a relatively minor mistake in the scheme of things anyway. Now is the time to do what it takes to meet your requirements, which the current code base sounds like is making mighty difficult.

    Since we can only deal in the here-and-now, I suggest, if possible, hacking it together with spit, twine, and duct tape. And getting your manager's buy-in for a clean up of all that code (a paying-down of the technical debt) to something more manageable, and, hopefully, more flexible.

    Good luck.

Re: what to do when you screw-up?
by BMaximus (Chaplain) on Feb 27, 2006 at 19:25 UTC
    I've been in a position where I've painted myself in to a corner and have had a hard time figuring out how go get myself out of it. I think you've already taken the first step in fixing the situation. You've taken ownership of your mistake and it shows that you care about your work. In a case like this it's not about blame. Explain to your boss why the program shouldn't be used and the ramifications if it is used as it is. Will the program take the server down if a lot of people use it? Does it give bad data or mangle it? Look at the program and if it does do what its been designed to do then maybe you need to look at the reason why you're not satisfied with it. If you're not the one setting the priorities then you need to take that in to consideration when you're setting out to refactor a program. I did it once, the boss knew that the program needed to be optimised and I was lucky in that part. If you decide to redo the program, since you already know what it's supposed to do and probably have a good idea on how to optimise it so its better more maintainable code, take the time to plan ahead on what you're going to do to reach your goal. Then to make an obscure reference from Thinner, "Eat your own pie."

    Update: Tanktalus makes a good point. Don't feel guilty and don't feel like you have to rewrite the entire program. Sometimes its best to let it be and when you have time to kill, that's when you rewrite it.

    BMaximus
Re: what to do when you screw-up?
by Anonymous Monk on Feb 27, 2006 at 19:47 UTC
    The best course of action depends on why your simple and straight forward program ballooned into an unmanageable monstrosity. Is it because you misunderstood the requirements when you started it? Is it because people wanted more features after they saw what you could do? Is it because you underestimated the complexity of the legacy code? Is it because the legacy code doesn't work as documented?

    To move forward, you need to fix whatever problem caused you to get into the situation to begin with. Once that problem is fixed, look at what code you've got now, and decide what needs to be done to get it where you want it.
    It doesn't matter who’s fault it is. Everyone makes mistakes, and as long as people aren't repeating the same mistake, going around placing blame is just a waste of effort.



    Regarding rewriting at home, I assume you mean your rewriting on your own time. Ultimately, the program belongs to the company, so even if you screwed up when you made it, its still their mess. If the program is going to be of value to the company, they should pay to fixed it. If its not going to be of value, they should scrap it.

    I know the feeling of wanting to be proud of your pet projects, but sometimes they just need to be scraped. If you offer to fix it for free, the value to the company no longer enters into the equation, so you might just be wasting your time on something that isn't really all that important anyway.
Re: what to do when you screw-up?
by Anonymous Monk on Feb 27, 2006 at 20:38 UTC
    Coding, in a purely academic sense, can take as long as it takes, and untold resources can be devoted to making sure it's correct, fast, and robust.

    In business, there are other costs: costs for your time, costs for missing deadlines, opportunity costs (eg. not having coders to work on a quick, high revenue project because they're all slowly and methodically refactoring old code) and so forth.

    Code management is a gamble: if you write a perfectly clean, shiny, maintainable codebase that never gets touched again, you've wasted the company's time and money on something it never really saw value from (code that could be maintained, but never was). On the other hand, if you write a horrible, unmaintainable mess, assuming it will be a one-off script, the rewrite may be very costly when business needs change.

    It's hard to predict what business requirements will be in the future. Some people advocate always writing the best quality code you can, at all times: this is a nice goal, but sometimes management won't let you do it. You have to respect this; sometimes your perception of the long term business risk won't match your employer's; and it's their money to risk, not yours. Let them know the cost/benefit analysis of a given decision, but accept their decision. There's no point in doing a major re-working of the billing system if the whole account collections department is going to be outsourced in three months. Sometimes management is right.

    So, give your management the most honest appraisal you can; but then don't blame yourself if you're wrong. After all, they knew the risks up front...

    Just my $0.02,
    --
    Ytrew

      Some people advocate always writing the best quality code you can, at all times: this is a nice goal, but sometimes management won't let you do it. You have to respect this; sometimes your perception of the long term business risk won't match your employer's;

      If only it were that easy. This is complicated by the fact that your reputation is created from the code you write. Even if it's management specifically tells you not to spend any time commenting/documenting/simplifying your code, its your reputation that is on the line when another programmer comes along and sees your name attached to the code.

      Your own minimum standard needs to be maintained, even when doing the work for others.
        Your own minimum standard needs to be maintained, even when doing the work for others.

        That's not always legal; when hired to do a job, you're really only legally allowed do what your superiors tell you. It's their money and their company, not yours. Using it to advance your own personal goals is unethical at best, and even criminal at worst.

        If you find it morally objectionable to do a certain task, you can always resign. I've done it ( in the middle of the .com bust, too). I didn't regret it, though money was tight for a while.

        --
        Ytrew

        Some people advocate always writing the best quality code you can, at all times: this is a nice goal, but sometimes management won't let you do it.

      Well, the 'best code you can' isn't an absolute -- it depends opn the resources available, like time, ability to talk with other developers and availability of test systems (one of my co-workers had to do Automated Teller Systems testing on a Production machine, taking real money out of a real account, then hand it all in at the end of the day). Oh, and there's another thing -- interruptibility -- if you're trying to keep your head down on a development job, but you *also* have to do support, double or triple your schedule. Or even tell your manager you can't do it unless someone else covers your support duties.

      Anyway, those are just a fwe things to think about.

      Alex / talexb / Toronto

      "Groklaw is the open-source mentality applied to legal research" ~ Linus Torvalds

        Well, the 'best code you can' isn't an absolute -- it depends opn the resources available

        Exactly; the best code that you can write, assuming infinite resources, is possibly (even probably) not the best code for limited resources.

        Oh, and there's another thing -- interruptibility -- if you're trying to keep your head down on a development job, but you *also* have to do support, double or triple your schedule.

        Lost focus costs time; at one job, I found that I was terribly unproductive, despite my best efforts, when I had a co-worker who cleared his throat, loudly and repeatedly, every five minutes. It wasn't really his fault, but it killed my ability to concentrate. Eventually, he was let go... I don't know if that was why...

        --
        Ytrew

      I cannot count the number of times that I've thought to myself while cleaning some trivial detail up, "Is this really worthwhile? Am I really going to care about this later?"

      However it seems that every time I do that, I find myself thanking myself for that cleanup several times over as the cleaned up bit repeatedly makes my life easier.

      YMMV. In particular experience has given me a fairly good intuition about what kinds of changes are useful, and what aren't. (I don't try to make everything a gem! Conversely I've seen a lot of people spend a lot of energy on counter-productive "cleanup".) But these memories are why I continue cleaning things up.

      You have to respect this; sometimes your perception of the long term business risk won't match your employer's; and it's their money to risk, not yours.

      The Crooked Timber Of Software Development

      Makeshifts last the longest.

        That sums up many of my feelings quite well.

        I've thought seriously about how to bring such professionalism to software development. Rather than try to teach people the "true meaning" of "professional", I talk of making real software engineers, like the engineers that build bridges.

        To bring engineering / professional practices to software development will require gradual steps. But I'd like those gradual steps to be made more visible so that people can notice the attempts and some can throw their support behind them.

        The basic concept is to start to encourage a small number of the best software developers to make the commitment of becoming true professionals worthy of being called "engineers". These few pioneers would be required to be trained in basic priniciples of software development that are so often neglected, including formal risk assessment, real encryption and security analysis, mathematical evaluation of software quality (such as statistical analysis of testing), and thorough testing methodologies. They would be ethically and legally bound to high standards as described in the article you linked to, in order to provide a "higher law" than "he with the gold makes the rules" that is so common in our industry today (one that finds no shortage of apologists, unfortunately).

        The role of these engineers would be to supervise software development, ensuring that proper techniques are used and ethical practices followed. Then market forces can be used to encourage the growth of software engineering. Software projects supervised by an engineer would be "certified" and customers could choose if they wanted to pay the extra price to buy the product that was certified vs. one that isn't. Once we have software engineers, the U.S. federal government would likely quickly come to require engineering certification for much of the software that it buys (especially projects with important safety considerations).

        And, it is my experience that I waste a great deal of my time as a software developer in having to deal with the shoddy software that I use. So I suspect that the market may realize that the engineered software can actually be the better bargain.

        I need to research how other engineering disciplines came to be. I suspect that at least two universities and two professional societies would have to join forces to create the first approximation to a software engineering program. I'd want accreditation of universities, licensing of engineers, requirements for education, continuing education, ethical standards, etc. all evolving through community review. All of this built up by the universities and societies setting these high standards and then government passing laws to give weight to the results.

        I also suspect that this won't happen until some tragedy where a lot of people die because of shoddy software.


        As for us peons... I'm shocked that so many software developers so consistently appear completely unwilling to demonstrate their expertise to whoever is paying them for their expertise. The choices aren't "do what the boss / client says even though it's a really stupid idea" vs. "tell the client how stupid they are and probably get fired".

        My reaction to "stupid" requirements is to discuss the requirements with those who set them (often taking a bit to find what the real source was) to understand why they think it is a good idea, explain why I, as a hired expert in the field, feel that it isn't the right choice (that is, to give them the benefit of the expertise that they are paying me for), and to help them to understand so that they can make a better decision(though not always a different decision).

        Most often, the stupid requirements were not properly evaluated. The first attempt at raising the issue usually gets a "that's the way it is to be" simply because my immediate supervisor didn't make that decision and so isn't the one who can authorize me to change it or because he helped make the decision and so it takes a small amount of explaining to get him to see things from a different angle.

        I talk to my supervisor and either he explains it better so I realize why it is a reasonable compromise, he understands my objections and the design is changed, or we realize that he doesn't have the full justification and I'm not just overlooking something which means that I move on to talk to the next person that is closer to the source of the decision.

        This sounds complicated, but it usually doesn't take long and it is just good communication that not only demonstrates that I'm taking the job seriously and that I have some expertise, but also almost always improves someone's understanding of the goal and very often improves some aspect of the design. Good communication makes a project more likely to be successful.


        My first "real" programming job was for a company of about one dozen employees. At one point management become concerned about people using our software without paying the proper fees to us. They informed us developers that we'd need to write a licensing system that would disable the software if lack of payment was detected. Two of us said it was a bad idea, a really bad idea. We explained that every license server we'd seen has problems, often serious problems, where people had paid but the software refused to work. We noted that people could die (due to the nature of our software) because of these inevitable problems with trying to make the software be able to cripple itself.

        Eventually, management responded that they'd heard our concerns but that it was still decided that licensing would be required. We each separately responded that we would refuse to write it on ethical grounds but also because it is would not be in the company's best interest based on our expertise. We wouldn't prevent other developers from writing it, of course. We were told that we'd be fired if we didn't write it. We mostly shrugged.

        The project to add licensing never materialized.

        Another time I was called in to help write up an official response to a proposal. I didn't understand one of the requirements. I asked about it and it became less clear to me why we'd want that and the others didn't appear to really understand the justification for it. I said we need to go talk to someone and get this clarified. I was told by my boss's boss that this was just the requirement and it was our job to fulfill it, not to question management's decisions. I politely noted that we couldn't do a good job if we didn't properly understand the background and left the meeting to walk to the president's office. I was followed by my boss's boss who was rather upset that I was being so rude and unprofessional.

        I explained my concerns to the president and he immediately responded that they hadn't considered that when they made the decision and the decision was changed, after less than 2 minutes of talking, and he thanked me for informing him of the problem.

        Just this week I had a rather small but complex feature requirement that was rather precisely if quite simply spelled out in the specification document. Unfortunately, the specification defined the only moderately complex rules that were to be implemented but made absolutely no mention of the justification.

        I did not just run off and implement what was spec'd out. It wasn't that I had any objection to the design. It was just that I had no way of validating that some simple mistake hadn't been made in recording the details of this part of the design and not enough ways to validate that I was interpretting the specification correctly.

        I spoke to the author of the specification document who said it was a complex decision, something to do with avoiding more expensive delivery options, and then forwarded to me the e-mail chain in which the decision was made. Unfortunately, the e-mail chain showed some give-and-take and evolution on what was decided, but again lacked any elaboration on the motivations.

        So I asked my boss about it and got a only few more tidbits. So I interrupted the ever-over-busy principle architect because he had been involved in the e-mail chain. He had some more parts of the justification. I bounced my new-found bits of info off the people I'd already talked to and got a bit more clarification. I did some digging into some rather complex moving parts that were required including talking to some database developers.

        In the end, I implemented exactly what was specified in those 4 short sentences of the design. I also included two pages of documentation in the code covering exactly why it was done that way.

        Oh, I also uncovered some flaws in the database logic that I fixed and forwarded my extended documentation to the parties involved since none of them had the whole picture. I got one more clarification in response. I think that was a job well done.

        By the way, yes, we are all quite busy and under dead-line pressure. And today someone new came across this same feature and was completely baffled at what the justification might be. A whole lot of time was saved because I just pointed them at my documentation. Personally, I'd have included the justifications into the spec document, but that wasn't my call and the existence of the my documentation (checked in with the code) is enough.

        Finally, there is another feature that just makes no sense to me. I've mentioned it to several people when the subject came up and they all report that it was discussed to death and that is just how it is. The feature stays the way it is. However, I now know who made the decision and understand some of the history behind it. So, it won't be long before I casually bring up the subject with the VP and founder (this is a much bigger company) so I can likely convince him that the decision used to make sense but doesn't anymore. And the product will eventually be improved because of this and the company will benefit in some tiny way...

        That's my job.

        - tye        

Re: what to do when you screw-up?
by blue_cowdawg (Prior) on Feb 27, 2006 at 21:19 UTC
Re: what to do when you screw-up?
by arcnon (Monk) on Feb 28, 2006 at 03:22 UTC
    This was a simple deal when I started the project it only had a simple object structure... 37 fields. Only over time has it now spiraled to right under a 1000. You couldn't edit the data entered in the original concept. No bells or wistles just data entry and a single report.

    after adding a feature here then a field there. Add some core functionality without breaking every other application in the mean time.

    Then somewhere in the middle certain users needed the ability to enter 'private data' which was added. Now the application has 4 tier levels of data entry. And the reports generated dynamically depending on the level of access now number 32.

    So I have slowly created the monster. And the worst part... the scary part is the application has been in use from almost the word go. Each upgrade has gone with out much down time 6 hours over the past year. I know I can easily it roll back but it is a butt clenching episode.

    now with needing to implementing it 4 times over. Trying to decide on creating "Evolutionary application trees" or try and maintian a communinal source base. Is a real toss up.

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: perlmeditation [id://533103]
Approved by GrandFather
Front-paged by planetscape
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others wandering the Monastery: (4)
As of 2014-08-31 06:19 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    The best computer themed movie is:











    Results (294 votes), past polls