|No such thing as a small change|
Re: OT: When to put things liveby jhourcle (Prior)
|on Apr 19, 2007 at 14:06 UTC||Need Help??|
Some of these issues have to do with your specific job, and the nature of the systems.
For instance, when I worked for a university, if a system change would require downtime, we never started before 6pm on a Friday for planned change, so we had the whole weekend to clean up if things went wrong.* Things were put into place on Friday night, stakeholders got to review the system on Saturday, and sysadmins and programmers had to get it working by 6am on Monday.
If it was something that could be done with a quick cutover, I'd prep everything in advance, get signoff on it in testing, and then cut it over at 7am on a Monday morning. (specifically because people came in late on Mondays, so the trouble calls came in slower**).
These days, my work is international, and as I'm a contractor, they don't like me working overtime or odd hours. So, I'm a firm believer in the Tues-Thurs window. We had a specific rule of NO system changes after noon on Fridays. For some types of changes, we have to wait for specific gaps which occur. For other changes, we do 'em late morning, as most of Europe's left for the day, and many of our West Coast users aren't early risers. We then have people around for a full day of debugging, should something go wrong.
So, I'd have to say my normal window these days is Tuesday-Thursday, 10am-1pm. (if it's a local change, and not going to affect the European folks, 8am-10am)
* the 6pm rule, unfortunately, is what got us into problems when our management wouldn't let us take down a mail server when we noticed it was having disk problems, and resulted in us spending 16 days around the clock trying to get things working (and by 'working' I mean e-mail was lost for all students) ... because management didn't want to react when we noticed the problem (a little past noon), and the system failed in a cascading manner at about 4:30pm.
** Of course, that also resulted in a problem one day when I got all of the signoffs, but something wasn't transfered cleanly ... and as no one noticed 'till about 10am, I had to manually merge changes, which took me about 2 days ... normally, not a big deal, but the cutover was just to buy me 2 weeks to finish a project ... which then spiralled out of control, and took years to complete (of course, I had been fired for 'use of sarcasm', as as the lead on the project, that might've explained why they were delayed by 3+ years (well, that, and bringing in a 'third party' to review the system, who didn't understand our business needs, or the software we were using which added the first year of delay, and may have resulted in my sarcastic attitude))