Re: OT: When to put things live

[Old_Gray_Bear]

That's pretty much the size of it. We aim for 'non-disruptive enhancement' around here. That means:

• No Friday installs of anything that has a Customer facing component.

  Of late this has become problematic, since we have a sizable User base distributed around the Globe. Someone is always inconvenienced, no matter when we install; so we have settled on Developer Convenience as the determining factor. This means that the last five or six Product releases have occurred on Wednesday Evening between 1800 and 2000.

• The Service DOES NOT GO DOWN.

  The New Service (Client enhancement, bug fix and code maintenance, new database features, etc) will run in parallel with the Old for a period of weeks. (Or a period of years, in the case of Client code. We have Client-software in the Wild that is over eight years old. The feature set is still supported, We don't accept bugs on it, however.)

• While there is a committee involved in the planning and scheduling of a Change, there is a single Change Captain. The Change Captain has final authority to say 'Oops. Back it out'.

• We try to give our User Community a reasonable estimate of when a new feature set will be available. We plan to have it in play four to six hours before the announced go-live. This gives us a little 'final-checkout' time. Our Users know this and so we sometime get 'early adopters'. We don't discourage this.

• The Usual Time Line:
  1. Two weeks before the date -- feature freeze
  2. One week before the date -- code freeze and QA begins regression testing.
  3. On the day:
     1. 1300 -- Final Change Review meeting -- are we really ready?
     2. 1600 -- New code/hardware active and checked out
     3. 1800 -- Go-time -- everybody involved gathers in a conference room and watches the logs and monitors.
     4. 1810 -- Pizza delivered (on the Project Managers nickel)
     5. 1930 -- More Pizza, this time with beer (ditto; there is a line item in the project budget for this)
     6. 2000 -- Go-Live for the Users; Ice cream arrives
     7. Afternoon of the following day -- Post Mortem

This seems to be a working method, it has served for the past fourteen months. We have only had one release aborted by the Change Captain -- when it was announced in the 1300 'final readiness' meeting that the primary power system to one of the co-location facilities had failed at 0300, and we were on standby generators. The power vendor 'expected to have it back online by close-of-business today.' The CC said "that's nice. We ain't going until the generator has been up for at least 12 hours." We slipped the install a day.