Archive for August, 2012

No Slam Dunks – An Update on the Qwest for IT Perfection

You may have recalled that way back in January of this year, I started a campaign within my company to eliminate self-inflicted technology outages. The slogan for the campaign was “There are No Slam Dunks in IT.” After a rocky start in January and February, our team got on a roll and posted an impressive 190 days without a self induced outage.

During this time we made numerous changes within our environment. We added new locations; we upgraded major network infrastructure within our data centers; we replaced telephone systems; we virtualized servers like crazy; and we did a major email/collaboration platform upgrade. Through it all we followed our steps of good change management. We planned. We communicated. We executed. We tested. We communicated. It was a beautiful site to see.

But I noticed something as the days started piling up. Slowly but surely the confidence of the team swelled. Unfortunately it swelled to a point, where it felt like as a team we began to think that there were indeed “slam dunks in IT.” It seemed the more success we had, the more corners we cut. We slid in extras during maintenance windows. We stopped doing some of the double checks we were doing earlier in the year. Our communications lacked sufficient details. We were still following our change management steps, but to some extent we started going through the motions. We reached a point of being too confident.

After a few near misses in early August, our streak of days without a self-inflicted outage came to an end last week and two days later we stumbled again. While you never like to have outages, I was almost relieved that we stumbled. I wish it had just been one stumble and not two, but I think the issues brought our team back down to earth and reminded us that we have to continue to be diligent in following our change management processes. It highlighted that going through the motions doesn’t cut it.

We have reset the clock, but I am confident that we can once again get on a good roll of days without IT causing an outage. I just hope this time around the confidence does not go to our heads because Slam Dunks aren’t allowed on this court!!