Aspirin of Stability

Scott wondered if one donut was going to satisfy his hunger as he filled his brand new super jumbo coffee mug with some fresh brew.

Phil’s eyes drift up from a pile of trouble tickets and focus on Scott, who walks towards him holding a silver coffee mug. Phil called, “Hey nice coffee mug, how much?” Scott responded, “the coffee was $1.75, but the mug was free”. Phil said, “How did you get such a great mug for free?” “Well”, Scott replied, "I won this mug at an EMC vendor raffle” Phil interrupted, “I cannot believe you let yourself become part of their propaganda machine!”

As the smiles faded from their faces, Scott sat and Phil continued, “As I was saying earlier the plan is to escalate the problem to the vendor”. Scott said “We’ve already have a severity one ticket opened with them for a month”. Phil replied, “I thought so too, but the vendor informed me that our ticket was closed until we load the latest patches”. Scott interjected, “I told them a month ago, there is nothing in the patch readme files to suggest that our problem is resolved in those new patches”. Phil continued, “That is why we need to escalate, we pay for a premium support contract, and as soon as we get a tough problem they read from a script, ‘load two patches and call me in the morning’. “

The problem that Phil and Scott are faced with is not unique. Since ENIAC, the world's first electronic digital computer, vendors have been pushing the patch as the aspirin of infrastructure stability.

Phil said to Scott, “In reviewing these trouble tickets I noticed that only about one half of the nodes ever had the problem, any ideas?” Phil went off to refill his coffee; Scott carefully studied the list of problem nodes.

When Phil returned, Scott announced, “These servers are also running application training, so somehow application training might also be related to the problem”. Phil replied, “Scotty, you found the needle in the haystack!” To that Scott answered, “But Phil, you found the haystack”. After some mutual back patting, they each grab their coffee and depart.

Over the next few days Scott and the application group identify and resolve the problem, which actually turned out to be an error in the application installation scripts. The problem had nothing to do with the vendor patch recommendation. If Phil and Scott had followed the vendor’s advice: their efforts would have wasted a lot of time, generated a tremendous amount of change, and have been totally unnecessary.

A few weeks after the problem was solved, Phil runs into Scott on the cafeteria checkout line and now Phil is carrying the same super jumbo environmentally conscious coffee mug. Scott asks, “Did you win that from EMC?” Phil smiled and said these words of wisdom, “The moral of this story Scott is that vendors don’t always understand our problems but they sure do have some great coffee mugs.”

See my related article in IT Managers Journal - The Patch Paradox

Copyright Dave Nocera 2007