You may have recalled that way back in January of this year, I started a campaign within my company to eliminate self-inflicted technology outages. The slogan for the campaign was “There are No Slam Dunks in IT.” After a rocky start in January and February, our team got on a roll and posted an impressive 190 days without a self induced outage.
During this time we made numerous changes within our environment. We added new locations; we upgraded major network infrastructure within our data centers; we replaced telephone systems; we virtualized servers like crazy; and we did a major email/collaboration platform upgrade. Through it all we followed our steps of good change management. We planned. We communicated. We executed. We tested. We communicated. It was a beautiful site to see.
But I noticed something as the days started piling up. Slowly but surely the confidence of the team swelled. Unfortunately it swelled to a point, where it felt like as a team we began to think that there were indeed “slam dunks in IT.” It seemed the more success we had, the more corners we cut. We slid in extras during maintenance windows. We stopped doing some of the double checks we were doing earlier in the year. Our communications lacked sufficient details. We were still following our change management steps, but to some extent we started going through the motions. We reached a point of being too confident.
After a few near misses in early August, our streak of days without a self-inflicted outage came to an end last week and two days later we stumbled again. While you never like to have outages, I was almost relieved that we stumbled. I wish it had just been one stumble and not two, but I think the issues brought our team back down to earth and reminded us that we have to continue to be diligent in following our change management processes. It highlighted that going through the motions doesn’t cut it.
We have reset the clock, but I am confident that we can once again get on a good roll of days without IT causing an outage. I just hope this time around the confidence does not go to our heads because Slam Dunks aren’t allowed on this court!!
Robert Not Bob 
Robert,
In the beginning, we train hard to build our confidence. Once we have several “wins” under our belt, our over confidence leads to complacency, and since we naturally won so many times, we no longer think we need to train. As a result, our shots become sloppy and our key performance indicators statistically diminish. So my question is how do you keep a confident IT staff motivated to perform the various drills necessary to be able to execute like a well oiled machine? Are you able to analyze the past 190 days and determine if there was a point where your team results began to diminish and ultimately lead to the outage? Did the team perform the same functions over the last 190 days? Would it be possible to rotate similar responsibilities between Techs over a given period of time to avoid complacency? I enjoy reading your blogs!
Robert – Good insights. I look forward to future thoughts from a person who has been there, done that, and still maintaining a learning perspective.