OT-ish: Estimating time to code

[kwaping]

I am really bad at coming up with time estimates for coding projects, and I'm getting asked to do that more frequently lately. Can you please share some tips or techniques you use for this?

[Tanktalus]

First, to calm your nerves, no one is "good" at coming up with time estimates for coding projects. Some are better than others, but the bottom line is that software is so unbelievably complex so quickly that getting it all in your head, determining pieces to be done, and figuring out how long that will all take pretty much takes as much time as actually doing it.

In fact, in my experience, it's nearly as difficult to tell how long something took after the fact (hindsight is 20-20? Not in my experience!) as it is to predict it ahead of time. So, don't fret that you're bad at it. It's not like you're unique at that ;-)

My rules:

1. NEVER underestimate. Don't be an optimist about how things will turn out. Everyone seems to overestimate their own productivity, and assume that nothing will go wrong. Both of these assumptions are always untrue. (Statistics say they must be right sometimes, but that's not my experience.) This is even more important when sizing things that other people will be implementing.
2. When I was first doing estimations, I'd use the 20-60-20 rule. Ok, I don't know if it was a real rule, but I was using it anyway. What it is is that I'd actually come up with three estimates. My optimistic estimate, plus a realistic one, and a worst-case one. I'd then assign weightings of 20, 60, and 20, respectively, and then take the weighted average. (I'd also usually provide all four estimates to my manager - and bold the weighted-average estimate as my "official" estimate.)
3. Details. The more detail you can come up with, the more accurate you'll be. Or, if you're missing something, the better chance a coworker will spot it and point it out. "You're missing the integration with the frobnoozer software - how long will that take?" Or they'll disagree with the individual sizings: "Implementing the braznaut feature couldn't possibly be done in less than a week!" "Why don't you just use the Standard Template Library? Should only take a few days that way."
   Also, I've found management to appreciate that I've already provided all the justification for the sizing, and they can just forward it to their management as-is. I've gotten a fair bit of airplay with upper management that way.
4. Don't forget: meeting about a feature counts as effort in producing it. 8 people in a 1-hour meeting counts as 1 person-day of effort. If you need a meeting to figure out the details, or to co-ordinate multiple people working on it, it's overhead, and needs to be accounted for.
5. Also, remember that learning new technology to implement something needs to be accounted for in the sizing. New people on a team will need some time to get up to speed - so they'll be 50-100% slower than their coworkers, and you'll need to have someone spend time working with them to bring them up to speed. The mentor's time needs to be in the sizing, too.
Above all - find a balance between time spent getting an accurate sizing and that accurate sizing. Don't spend two weeks figuring out how much time a project will take when it should be in the 20-30 hour range. On the other hand, something that will take a year to do deserves some extra time spent on it up front.

I've learned to do fast sizings, and I throw in a fudge factor on everything. Usually 25-50% (it used to be closer to 100% until I started getting more accurate on my thumb-suck guesses). These are all just guesses anyway, so a fudge factor seems appropriate ;-) This helps account for things going wrong, for meetings, and other details that I keep forgetting.

Update: last rule: I always round up. Anything over 4 hours becomes 1 day. Anything over 3 days becomes one week. Anything over 3 weeks becomes 1 month. Anything over 7 or 8 months becomes 1 year (let's face it, we're looking so far into the future that it's pure bunk anyway). Anything over 2 years of effort is beyond what we can reasonably do by the next release anyway, so it becomes moot. I only do this rounding after having broken down the problem, sizing the pieces, and adding it all back together. If each piece is 3 days, and there are four pieces, I don't round each one to 1 week, and come out with a month. I add it up as 12 days, which is 2.4 weeks, and round that to 3 weeks.

Good luck!

[Jenda]

Just a very simple tip. After you do the estimate, write it down. After you acomplish the task, write down the hours it actually took and the things that you think took longer than expected or those that you overlooked when estimating. Keep this document and add to it all your estimates. It's good to be able to see later whether you are getting better and how much do you have to change your estimates and when doing other estimates you have a ready list of things you tend to overlook.

[adrianh]

I am really bad at coming up with time estimates for coding projects, and I'm getting asked to do that more frequently lately. Can you please share some tips or techniques you use for this?

My rules of thumb are:

1. Any individual estimate over three hours is a guess that can be wildly incorrect. Split or simplify tasks until they're all under three hours.
2. Try and keep all of your tasks about the same size. It makes moving tasks around and re-prioritising so much easier.
3. Look at how many tasks you did last week. Assume that's how many you'll do this week. Adjust milestones appropriately. (aka Yesterday's Weather)

[renodino]

I suffer from "programmer's optimism": I invariably underestimate how long a project is going to take.

And so, after 25 years experience, I've come up with a pretty reliable algorithm for computing schedule estimates:

1. Take the "instinctive" time I think I can do it.
2. Multiply it by 3.
I've been very surprised at how accurate that simple algorithm has been. Whether its just a week long project (which my gut always tells me will take "2 days, tops"), or a 12 month project (which my gut always tells me is a 4-5 month effort).

YMMV, of course. Your multiplier may be different. And if its going to be a team effort, that may effect your multipler as well (tho, inexplicably, I've found the 3x factor seems to work in that case too)

Presumably, you've concluded your estimation skills are flawed based on prior experience...so take the estimates you made in those cases and compare to how long the project actually took. You may be amazed to find the same result I did: a simple multiplier, regardless the duration/size of the project.

[tilly]

I highly recommend Software Estimation: Demystifying the Black Art by (of course) Steve McConnell.

While you're at it, I also recommend everything else he has written. :-)

[ptum]

I also am very bad at coming up with time estimates. Whenever possible (and when working with internal customes), I try to sidestep the issue, by promising a 'working prototype' much sooner than the customer could expect a product. Sometimes they are so delighted with this promise that I can avoid having to give a time estimate altogether. While they are picking the prototype apart and wallowing in scope creep, I quietly code the parts I didn't have time for, and then start giving them pessimistic estimates for the features I don't like and reasonable estimates for the features they really need.

I think in some organizations, the request for 'time estimates' hides two more basic questions:

• Am I going to have to wait a long time to get this functionality, such that I'm better off getting someone else to do it?
• Is the cost of this software going to be greater than its value to the company?

By quickly producing a 'working prototype', I alleviate fears about the first question, and I'm in a much better place to address the second question once I've shaken the real requirements out of the trees. Sometimes you build the prototype and discover that the need really isn't there ... some time is wasted, but usually not too much.

This works best, of course, in an organization where a certain amount of autonomy is granted and where there is already a trust relationship established, and it pretty much sidesteps your original question, but, heh, it is the technique I use. :)

[apotheon]

This reads like a gentle, and informative, version of my own answer: avoid giving time estimates if at all possible. One way to do it is through a sidestep like you've provided, which involves some initial prototyping and requirements "bargaining", and coding on the side when they aren't looking to "sneak" in some work before the estimate (thus making a substantial part of your "estimate" an actual real-world measurement of the work you've already done). Another, better (in my opinion) but far less likely, approach is to actually finagle your client/boss/whatever into adjusting to suit a business model that isn't dependent upon time estimates.

I'm not advocating something like eXtreme Programming or Scrum Agile Development. I'm talking about telling the employer "I'll provide you with a working prototype. If you like it, I'll work up something that is actually usable for what you need, and you'll pay me for the work so far. If you like that enough, I'll start working on feature requests and get paid for them as I finish them. If not, you can get someone else to add features, since you'll already have working software that can be extended. We're all happy, since I've gotten paid for what I did and you'll get the development you want, one way or another, without substantial wasted time or cash. Nobody gets ripped off."

While this can be adapted to salaried employment and/or product-for-sale development, in at least some cases, it is obviously best suited for internal tool development, web development, and so on, by outside consultants. It draws somewhat on the ideas expressed in the Agile Manifesto without falling into the trap of rearranging deadlines and making your death-marches and schedule overruns different rather than nonexistent. I've noticed that most of the formalized agile methodologies seem to turn death-marches into death-races while making it appear at first blush that they solve the death-foo problem altogether, which seems almost as suboptimal as just going all the way back to waterfall development.

Independent Internet startups (when they're done "right") seem to have the right idea, much of the time: eliminate formalized scheduling altogether, and just do things as quickly as possible without sabotaging the project. This forces prioritizing more effectively (get the most important stuff done first, the other stuff when you have time) and tends to eliminate the real hazards of over-schedule, over-budget development (a common result of having predetermined project-completion standards with project schedule estimates). This tends to work best for independent Internet startups because they get to do things however the hell they want to. Next best, for consultants who have the luxury of choosing their clients intelligently and have the social acumen to talk clients into it, because they get paid for the work accomplished rather than the time spent working. Third, for salaried employees who are working on internal tools or web development projects. It tends to work least well for people like Microsoft employees, because death-march development is almost indivisible from market dominating shrinkwrapped retail application development.

I left a category of development out of that explanation that bears special attention: noncommercial open source projects. The reason I left it out is that there's exactly zero perceived necessity of offering schedule estimates. The reason it bears special consideration is that open source development so naturally uses development technique similar to what I described, with each developer working on the features that are most important to him or her, with each project founder creating a barely-working prototype that gets augmented as opportunity arises because (s)he wants to start using it as soon as possible, that it serves as an excellent model for how to attempt to organize your development methodology in general. The only exceptions are that A) there's barely any motivation to work quickly at all after the first working prototype is developed, though development still often happens quickly, and B) development tends to end up far more distributed and less traditionally "project" oriented once the feature addition phase of development begins (though in some cases specific feature additions might be projects all their own).

Interestingly, Google's internal development process very nearly mirrors the general open source development community's processes, but with more focus in a sort of insulated world-in-microcosm developer ecology. As commercial software development becomes more service-oriented and less product-oriented in the future (and it definitely seems to be doing so), I think Google's example will probably become the canonical development process prototype for success. Time will tell, of course.

Unfortunately, most developers are still locked into a situation where the only way to get work, get paid, and hold down a job, is through coming up with development schedule estimates that aren't too egregiously off-target (they'll always be off-target, but they needn't always be egregiously so — such estimating is akin to voodoo). There's a lot of good advice in the other responses to the OP; hopefully it will be useful. My own post probably won't be of much use, unless you're in the rare (and lucky) position of being able to avoid schedule estimates altogether by constructing a software development business model that doesn't suck.

When doing web development, I can usually get close to what I describe, but ultimately have to give fuzzy estimates on certain project phases. It's not perfect, but it's better than trying to set a schedule in stone before the requirements are even hashed out (let alone altered later, as they always are). I think in that respect my circumstances are better than most, but not as good as I'd like, when it comes to project scheduling.

[BrowserUk]

This may be useful to you.

[robot_tourist]

I think, as other posts have pointed out, the best thing is to use your experience to really study/analyse the requirements, and possibly even to do a good design. Sorry, the _best_ thing to do is actually get good requirements. I once waited a couple of months after a verbal spec for a short project and got a one-page hand-written flowchart. Although that's partly my own fault because I said it was the minimum I would accept before I started work on it.

My other peeve is that I often get caught at 2pm on a Monday when I've just come in from an hour's 5-a-side-football to give estimates on fixing some change requests. One day I'll get on that assertiveness course. Until then I generally play it safe and only say I'll do a couple of small things for a given week. Which often works out to be quite accurate.

If your projects do slip, make sure your boss understands exactly why, especially if you have done a detailed plan. As I work supporting software for manufacturing tests (much of which was written by people who left before I joined the company), project timescales for new things will have to include some time out to fix unforseen urgent change requests for old code, or even for the slippage of lower priority change requests. That is never easy to account for. My boss isn't a software guy, but he does understand when projects slip for good reasons.

[radiantmatrix]

I use a mechanism I call "auto-padding". It works like this:

1. Break tasks into the smallest reasonable chunk you can -- if you can't measure the effort required in under a few hours (I use 6), it's not broken up enough.
2. Estimate the number of hours for each task.
3. Add up all the hours.
4. Now, define "a day" as 6 hours: divide your total hours by 6 to get the number of expected days:
   • Always round up
   • For projects longer than a few weeks, define a week as 4.5 days: divide your day count by 4.5 to get the number of weeks
   • When you have your weeks or days number, but you must supply an estimate in hours, use the business standard values and multiply back out: if you have a weeks number, take $weeks*40 (std 40-hr week); if a days number, take $days*8 (std 8-hr day).

Breaking your project into smaller tasks helps make your estimates more realistic.

Defining days as 6 hours and weeks as 4.5 days (of six hours each, note) gives you an automatic fudge factor that takes into consideration much of the unforseen events, meetings, vactions, sick days, and so on.

As an example, say I have the following task estimates:

Task	Hours
Implement foo	5
Implement bar	6
Implement foobar	3
Implement barfoo	6

I start with 20 total hours. I divide this by 6 to come up with 3.33 days -- I never give estimates in partial days, so I round this up to 4.

I've found estimating in this way to be reasonably accurate, as it accounts for "productive hours" rather than "work day hours". As a result, I often get finished a little early: in the above example, there's a good chance I'd finish a day early.

This gives me an opportunity to do more testing, clean up one or two things I thought were ugly but "good enough", etc. It also means I work a 40-hour week (usually), can take a leisurely lunch and regular typing breaks, and am able to spend some time reading PerlMonks -- but I still turn my work in on time.

[apotheon]

You should write a script to do all that arithmetic for you, with a pretty interface, and stick it on Sourceforge.

[astroboy]

Hear's another thread on this topic. As mentioned, you can download Steve McConnell's Estimation Tool

[ChrisCantrall]

Check out Painless Software Schedules by Joel Spolsky, March 2000.

I am also terrible at estimates, and I tried his method for my latest project - it really helped.

Additionally, I track my time in a seperate sheet in Excel, breaking up my day into generic tasks - Meetings, Useful Meetings, various projects, Administrivia, Helping the New Guy, etc. So I end up with 2 views of my time - a long-term tracking of my time by day and how I tend to spend that time, and a per-project estimate of how I think I'll be spending my time in the near future.