in reply to
OT: When to put things live
That's pretty much the size of it.
We aim for 'non-disruptive enhancement' around here. That means:
- No Friday installs of anything that has a Customer facing component.
Of late this has become problematic, since we have a sizable User base distributed around the Globe. Someone is always inconvenienced, no matter when we install; so we have settled on Developer Convenience as the determining factor. This means that the last five or six Product releases have occurred on Wednesday Evening between 1800 and 2000.
- The Service DOES NOT GO DOWN.
The New Service (Client enhancement, bug fix and code maintenance, new database features, etc) will run in parallel with the Old for a period of weeks. (Or a period of years, in the case of Client code. We have Client-software in the Wild that is over eight years old. The feature set is still supported, We don't accept bugs on it, however.)
- While there is a committee involved in the planning and scheduling of a Change, there is a single Change Captain. The Change Captain has final authority to say 'Oops. Back it out'.
- We try to give our User Community a reasonable estimate of when a new feature set will be available. We plan to have it in play four to six hours before the announced go-live. This gives us a little 'final-checkout' time. Our Users know this and so we sometime get 'early adopters'. We don't discourage this.
- The Usual Time Line:
- Two weeks before the date -- feature freeze
- One week before the date -- code freeze and QA begins regression testing.
- On the day:
- 1300 -- Final Change Review meeting -- are we really ready?
- 1600 -- New code/hardware active and checked out
- 1800 -- Go-time -- everybody involved gathers in a conference room and watches the logs and monitors.
- 1810 -- Pizza delivered (on the Project Managers nickel)
- 1930 -- More Pizza, this time with beer (ditto; there is a line item in the project budget for this)
- 2000 -- Go-Live for the Users; Ice cream arrives
- Afternoon of the following day -- Post Mortem
This seems to be a working method, it has served for the past fourteen months. We have only had one release aborted by the Change Captain -- when it was announced in the 1300 'final readiness' meeting that the primary power system to one of the co-location facilities had failed at 0300, and we were on standby generators. The power vendor 'expected to have it back online by close-of-business today.' The CC said "that's nice. We ain't going until the generator has been up for at least 12 hours." We slipped the install a day.
I Go Back to Sleep, Now.