Chapter 9: What Is That Light at The End of The Tunnel? – The ITSM Iron Triangle


I chugged the energy drink, tossing the empty into the trash and grabbing another in one continuous motion. I held the fresh can’s cold surface against my forehead, letting the cool drops of condensation trickle down my face, as I yawned. It felt good. But I was still foggy from the lack of sleep. I’d been working all day, and most of the night, for the last two weeks, and it was catching up with me. Nothing seemed to work to keep me awake anymore.

I was in my spare bedroom; the place I euphemistically called my home office. At least that was the idea I once tried to sell to the tax man … unsuccessfully, I might add. The clock on my desk added a cool glow to the darkened room. It was nearly 2 am, and my laptop’s screen was filled with data. I’d never be ready for my meeting with Ramesh in the morning, but it was my own fault.

I’d been able to build a database containing every incident for the last two years, and by attending every Sev1 response meeting, I was able to collect and add data on the new ones as they occurred. I’d spent the last day and a half going through each of the incidents individually, trying to figure out a way to improve our handling of incidents, so they would simply go away.

I couldn’t call what I had information. It would take me all night, and most of tomorrow, just to go through the rest of the detail. I’d been trying to summarize it for hours, but no matter what I tried, it failed. I’d charted and created tables, and means and distributions, but the more I tried, the worse it got.

There was still too much data. Sure, we’d been able to nail one of those repeating incidents and everyone was happy about that. But if I had to do that for all of them, and every time there was an incident, I would never make it. I was going to need three or four more people just to stay even.

I needed some feedback on that idea before proposing it to Ramesh, so I sent a text to Sean, asking if he’d meet for breakfast in the morning. I hoped he’d see it in the morning and respond, so I was surprised when I got a response from him almost immediately asking, “When and where?”

I had just chugged the energy drink and was washing the taste out of my mouth with a big swig of black coffee, when Sean walked into the restaurant and over to my table. He sat down and shook his head.

“You keep doing that, and you’ll rot your stomach, or have a stroke. Maybe both at the same time.”

“More job opportunities for you.”

Sean laughed as the waitress walked up.

“Morning, Hun,” she said. “You want a breakfast, lunch or dinner menu?”

Sean stared at her with a puzzled look. “Do you serve dinner at 6 am?”

She tapped her pad with the eraser end of her pencil. “Look, Hun, we have cops, fire-fighters, third shifters, truckers and computer geeks in here all the time. Who knows what time their internal clock is set for? We serve anything, anytime. And I always offer the choice to those who look the part.”

“And so who do we look like,” I asked with a grin.

“I tell you what, Doll. You go and look up ‘computer geek’ in the dictionary and I’ll bet they have your picture there staring back at you.”

Sean laughed and said, “Breakfast for both, please.”

“None for me,” I countered. “I’m good with coffee for now.”

“Aren’t you eating?” Sean asked.

“I’m afraid it will make me sleepy. I’ll get something later.”

Sean poured two creams and four sugars into his coffee. “So what was so important you needed to meet before work today?”

“You know that I was able to identify some repeating incidents all caused by the same thing, right?”

Sean nodded as he stirred his coffee, then took a sip. “That was a great piece of work. You should be really proud of that. Don’t know how we all missed the connection, but I’m glad you had the insight to spot it.”

“Do you know how many incidents I had to read and understand to do that … over 400. It took me weeks to find that one repeater. I’ve been going through the data looking for others, and everything is all jumbled up in my head now. I can’t keep them straight. There’s just too much for one person to handle. I could never keep up going forward. I can’t even get through all the ones we have. If we are going to do this, it will take more than one person … probably three or four. Either that, or else I’m going to need to call meetings with the technical teams a couple of times a week, so they can explain their notes.”

Sean shook his head. “You’re kidding, aren’t you? We have a hiring freeze on right now. You may have Jessica’s attention at the moment, but your boss will never agree to it. Neither will Jessica. Think it through. If you get a bunch of heads, everyone else will go crazy demanding their own, too … and for projects that can show a much more concrete ROI that what you’re doing. And don’t even think about going back to calling a couple of meetings a week about this stuff. Even I won’t attend. You’re delusional from too little sleep.”

“Okay, so what should I do?”

Sean shrugged. “If I knew, then I’d be doing it. But then again I wasn’t stupid enough to volunteer for the job without first knowing how I was going to get it done … like some people.”

I drained the last of the coffee in my cup. “You know that I was assigned this task because Sarah failed. Maybe you’re next in line for the job if I fail. So if you don’t have an idea of what to do, then you’d better get thinking.”

Sean shifted uncomfortably in his chair. “Hmmm, that may be true.”

I was a little pleased with myself. Up to this point, I had always let Sean box me into a corner, and by giving him back as well as he gave me for the first time, I was actually able to plant the thought in his mind that he had a stake in solving this, too.

The waitress interrupted us with Sean’s breakfast. She pointed at me with her pencil and asked, “So Hun, are you gonna eat something or not?” She offered me the Breakfast menu. “Breakfast is really good here. I recommend protein if you’re trying to stay awake.”

I flipped the menu open. There were at least six pages of breakfast items. There must have been 100 different items. There was every kind of meat I had ever heard of, everything from buffalo sausages to turkey omelettes. There were eggs done more ways that I imagined possible, and the list of pancakes, waffles and crepes seemed to go on forever. It was overwhelming.

“I can’t possibly decide,” I mumbled. The lack of sleep was fogging my brain. “There are too many to go through. It will take me forever. Why do they make it so complicated?”

She snatched the menu back from me. “Sure you don’t work third shift, Darlin? This ain’t complicated at all. It only takes a minute to pick something out. Let me show you how. First, do you want Breakfast, Lunch or Dinner?”

“"Uh, Breakfast I guess.”

“Okay, then you want the breakfast menu.” She handed the menu back. “Now, do you want mostly protein, carbohydrate, or sweet? Lookin’ at you, I’d head to the protein.”

“Okay, protein I guess.”

“Are you vegetarian?”

“Nope. I eat pretty much all types of things.”

“Low cholesterol or not?”

“Mine is already low. Lucky genes I guess.”

“Something exotic, or plain old comfort food?”

“Nothing fancy. Comfort would be good.”

The waitress fanned the menu open and shoved it into my hands. “You want something from this section, Hun.”

There were eight items in the section titled, “Classic Comfort Breakfasts”. Each was a breakfast staple and all met the requirements I had given her. I pointed to the bacon scrambler with smoked Gouda. “I’d like this, but I think I should have something sweet at the end.”

Without looking up from her writing, she said, “No sweat, Darlin’, just flip to the section called Classic Sweet Endings. It should be two pages further in.”

She was right, and it had six items. I chose the chocolate croissant.

The waitress smiled and said, “I’ll be back in a few with your food, Darlin.”

As she was walking away, Sean started to laugh.

“What’s so funny,” I asked, afraid he was laughing at me because I had so much trouble ordering breakfast that the waitress had to help. “There were hundreds of things there. It would have taken me forever to go through them all. And then deciding which one I wanted would have been impossible. There were too many things to handle.”

Sean shook his head. “You’re too close to it. You don’t see that she just showed you how to solve your problem about making it easier to uncover repeating outages.”

“I order breakfast for everyone?” I was completely confused and so very tired.

“What did she do to make it easy for you?”

“She asked me some questions about what I wanted. Any idiot can do that.”

“And what did she do with that information?”

I shrugged my shoulders. I really hated these little games Sean loved to play.

Sean reached out and motioned to a server. “May I please see the Breakfast, Lunch and Dinner menus? I think my friend is rather hungry.”

The server returned in a moment with the menus. Sean laid them side by side on the table.

“The first thing she did was ask you what meal you wanted. You said breakfast.” Sean set the other menus on a chair beside him. “That reduced the number of items by two-thirds.”

Sean opened the menu. “Then she asked you if you wanted something from the protein, carbohydrate or sweet.” He pointed to each of these sections inside the menu as he spoke. “You said protein.” Sean then folded the pages of the menu so that only the protein offerings were visible. “That reduced the number of items by two-thirds again.”

Before Sean spoke again, I saw the vegetarian, low cholesterol sections under protein. Looking a little further down the list, I saw a section called “Classic Comfort,” with its eight items, one of which had been my choice.

“Get it?” asked Sean as he fanned through the pages of the menu. “You need to segregate the incidents into categories and only consider them one category at a time. Then you’ll have small enough numbers that you can get your hands around them. Each meal has a number of category attributes to it.”

“So we can tell people how many outages were sausage and how many were eggs?”

“Something like that. You need to come up with categories that make sense for IT. Start with People – Process – Technology. Fit every incident into one of those categories. Then fit every incident into another layer of categories about what was going on when it happened. Use something like, making a routine change, installing something new; things like that. Or maybe even nothing. The incident trigger happened all on its own.”

It all snapped into place. It almost made too much sense. “And then I could add the team that either caused, or could have done something to prevent it from happening. It’s like a database with each event as an entry. And each of those entries has attribute fields … one for each category. So I can sort and group the events by category. Based on that, I can identify trends by area of interest and impact.”

I started sketching designs and making notes on the napkin, not realizing it was cloth, not paper. “I’ll want to get with our Mia and her DBA team to see if they can adapt the ticketing tool to do this. And if not, maybe they can build me a relational database that each of the technology teams can use to enter … ”

Sean cut me off. “Ease off on the caffeine, Chris. You’re getting ahead of yourself. This is not the time to boil the ocean. Keep it simple and small. If you can’t quickly build a workable tool using a spreadsheet, then you’re making it too complicated. Perfection is not required. All you need is some improvement.”

Sean took a sip of his coffee and added, “Just be cautious about pointing fingers, because you’ll drive people away. It’s about all of us getting better through continual improvement, not about achieving instant satori.”

I nodded. It all slipped into place. I just had to figure out which categories made the most sense, and remember that any event has multiple categories. The biggest risk to any work is the fear to step back and get a broader view of what’s important and what needs to be done. Being too close to the details, or too invested in the way things were always done, makes continual improvement that much harder.

I was so excited by the breakthrough, that I immediately sent a text to Ramesh telling him I was sick in bed and we’d need to reschedule. I used a text so I wouldn’t have to actually speak with him. I couldn’t have him finding out where I really was.

“My apologies, Sean, but I’m going home to work on this right now,” I said as I stood up and pushed my chair in. “Thank you so much for all your guidance. You really showed me a breakthrough.”

I put 20 dollars down on the table. “Here, this should be enough to cover your breakfast and the tip.”

Sean started laughing at me. “You are insane … in a good way, I guess. Best of luck. Remember, keep it simple.”

I nodded and walked away. I didn’t care if he was laughing. Now I had a way to boil the ocean … one category at a time.

Tips that would have helped Chris

Ideas for solutions will come from the strangest places. Don’t be afraid to ask for input, or consider non-IT solutions that could be adapted to your situation. Elements of the ITSM process can be seen in other businesses, and other industries. Don’t discount inspiration because of its source.

Solution inspiration often comes in bursts. It rarely comes a little bit at a time. Try to strike a balance between digging deeper and trying something different. People often confuse effort and results; thinking that working harder and longer on something will produce results, when in fact, they have already gleaned everything they can use from that source and need to move on to another venue.