Chapter 12: When No One Is Around – The ITSM Iron Triangle


I chugged the energy drink, then tossed the empty into the trash, and grabbed another from the break room fridge in one continuous motion. I held the fresh can’s cold surface against my forehead, letting the cool drops of condensation trickle down my face as I yawned. My eyes hurt, and I was having trouble focusing on the laptop screen. I knew you weren’t supposed to rub your eyes, but I did anyway. They were so dry, it felt like I was grinding sand into them.

It was nearly 2 am. I knew my laptop would be waiting for me back in my cube; its screen still filled with too much data, all waiting for my analysis. I looked at the vending machines to see if there were anything to eat that wouldn’t make me sick, but stepped away when I realized I was just trying to avoid going back to work. My back ached as I stretched before walking out into the hall. Here and there, cubicles had lights on, but there was no way of knowing if that was someone still working, or just a lazy attempt to make their manager think they were working late.

I was really beginning to resent the constant struggle to sustain even some level of effective participation by the IT leaders in my problem management process. Every step away by them meant that I had to do that much more. At times I was way beyond driving the root cause of incidents. I was actually trying to help the technicians work through their issues. That was the only way I could get them to participate. They respected the commitment on my part, even if I didn’t know that much. Maybe my being there just gave them someone to feel smarter than. But it was still wrong on so many levels.

To get the oxygen flowing in my brain, I took the long way back to my cube. A little exercise was just what my body needed tonight. The walk took me past the data centre.

The data centre had an unusual construction; expensive, too. A non-negotiable request from a previous CIO during the data centre build-out was for transparent walls along one side, so that they could walk guests by and impress the unknowing with all the rows of blinking lights, and the sterile look of the space. Very inefficient from a facility standpoint, but who knew what pressures the CIO was under at the time.

I took this hall many times during the day, and the data centre had always been devoid of people. But that made sense. The business had insisted on a standing rule of no changes during their prime hours of 7 am to 8 pm. And with the new customer relationship management system, Mountain Top, rolling out to the field in two days, they’d insisted on a total freeze, except for emergency repairs. Everyone had been sent a copy. The memo was posted on almost every wall, and even on the doors to the data centre.

I stopped when I noticed people inside the data centre. At first I thought they might be operators loading a tape or something. When I saw three people sliding a server into a rack, pulling floor tiles, and pulling wires up out of the floor for power and connection, I knew something was wrong. I checked my phone, but there was nothing about an outage. I swiped my badge on the door card reader, and to my surprise, the door clicked open. That shouldn’t happen. I shouldn’t be allowed inside unescorted. As I stepped into the mantrap, I wondered how many other people had unrestricted access.

A moment after the outer door sealed, the inner door opened. I stepped into the data centre, as a rush of cool air pushed up through the floor from the plenum underneath. Now I could hear laughter and voices. They sounded only a couple of rows to my left. I recognized the voices. It was Nidal, Roger and Shelia. They worked on the development team. I’d met all of them one time during an employee after-hour’s party. At least they were laughing. It’s good to enjoy your work, I guess.

I reached their aisle just as they locked the new server into the uppermost space in the cabinet, and began snapping power and network cables into the back.

“Hey, Chris. Great timing. You show up just as we’re finishing up our work. If you’d been here a few minutes earlier, you could have given us a hand to get this server in, so we can get loading the user interface for Mountain Top. You have the timing of a manager, that’s for sure.” They all laughed and kept working.

I shook my head. “What are you guys doing here? Why aren’t the data centre folks racking and stacking for you?”

Shelia giggled, “Cause they’re lazy. Getting them to do what it takes to get the job done, is like pushing a string. They don’t care a whole lot about meeting the needs of the business. The only time line they work by is their own. They’ve got so many work rules and schedules, that it’s worse than working with a bunch of paper-pushing bureaucrats. They have no concept of the word urgent, and don’t seem to care that this application needs to be functional for the business in two days.”

“Yeah,” interrupted Nidal. “They’re a bunch of bureaucrats who never want to step up and go the extra mile when the situation demands it. And they don’t understand what it is they are putting in. They’re not the ones who slaved for the last eight months over the code on this server we’re installing.” He pointed to the three of them. “Look at us. Here we are making sure this gets done, even though we really should be home in bed asleep. But we do this because that’s what it takes for the business to be successful. We’re the kind of IT people that you can count on to deliver … the kind that make IT a good business partner with the business. The company needs more people like us … people of action, and a lot fewer zombies like the ones that work third shift in the data centre.”

“You mean this is critical to Mountain Top going live in two days?”

Roger nodded, “Unless of course you’d prefer it not to work. Back-end functionality isn’t much good to the average salesperson without some front-end GUI.”

I couldn’t believe what I was hearing. I’d sat through the change management meeting three weeks ago, where the installation of Mountain Top was reviewed and approved. The leaders from the development team assured us that the application had successfully passed all unit and integrated system testing. They told the change management board that all development work was completed. Even the manager of these three had been there, and assured everyone that the GUI was ready to go.

“But what about the change management meeting a couple of weeks ago?” I said.

Roger shrugged. “Who knows? I wasn’t there. I just know we had to get the Mountain Top roll out approval from change management by that date, so we could support the business. But why the inquisition, Chris? You’re the one always talking about how we need to support the needs of the business. We did what we had to do, so that the business can move forward in two days. Isn’t that what a good IT business partner does?”

So the input from the change requestors had been all smoke and mirrors. They’d lied their way through the meeting in order to meet the deadline the business had set.

“But how did you get this change approved? Was it okayed as an emergency change?” I asked, afraid that I already knew the answer.

“We don’t need a change to install our own servers in an empty rack, right gang?” Nidal and Shelia nodded.

“There’s no question it needs to be done ASAP. So that makes approval moot,” said Shelia, as she snapped the last cable into the back of the server. “And if we’d asked, there was a chance some bureaucrat on the change board would have found a reason to make us put it off. They’re not accountable for keeping the company going. All they have to do is keep their chair warm.”

Shelia slapped the side of the rack. “Besides, we’ve installed servers a hundred times before. Any 12 year old can do it. It’s the only server in the rack. There is no risk here. And we’re doing it for the right reason … because the business demands it.”

She turned to Nidal and asked, “Shall we fire this puppy up and see if it lives?”

“Hell yeah,” said Nidal, as he ran his hand across the power switch. “I validate this code load so I can go home and get my life back.”

“But what about integration testing? Did you test this final version with the rest of Mountain Top? What if it impacts the rest of Mountain Top?”

“Don’t be such a worrier,” said Nidal. With a wide swing of his arm, he slapped the power switch to on, and said with a giggle, “It’s alive! It’s alive!”

This was very wrong. I started to say, “Stop,” but the words never made it out of my mouth.

“We did it,” said Nidal, as he leaned up against the server and slapped his hand against the rack really hard. Roger yelled, “Oh, shit … ,” as the entire cabinet, server and all, began to tip over directly at me.

In their haste, the trio had grabbed the only empty cabinet they could find. To protect the data centre they must have thought. What they didn’t know was that they had grabbed a cabinet that had simply been staged on the floor. It wasn’t yet bolted down and with the server incorrectly bolted in at the very top, the entire cabinet was unstable.

The toppling cabinet knocked me backwards and I watched in what seemed to be slow motion as Nidal’s hands reached around the rack, trying to keep it from landing on me. Shelia and Roger both grabbed for the sides, but couldn’t hold on. My rear hit the raised floor hard, my back slapping into the cabinets in the next row just an instant before Nidal lost his grip and the rack slammed into the cabinet behind me. The cabinet behind my back shook and tilted. Sparks flew as cables were severed, but it remained upright enough to prevent the loose rack from crushing me.

“Are you okay,” said Shelia, as she peered at me squeezed between the fallen cabinet and the one behind my back.

“As far as I can tell, other than some bruises, the only thing injured is my dignity.”

“Thank God,” she replied.

I heard Roger’s muffled voice in the background.

“We’re going to go … go and get help,” said Shelia, her voice sounding very tentative. “Don’t worry; you won’t be in there long.”

I heard feet running away as Nidal and Roger’s voices faded. I tried to extricate myself, but the weight of the cabinet was too much for me to lift from a sitting position.

There was a moment of quiet, broken only by the sound of air flowing up from the plenum beneath me, before strange new voices came near.

“Hey, there’s someone trapped under the rack. Come on, give me a hand.”

The cabinet began to rise as an unfamiliar woman with dreadlocks, and an ear full of piercings, squatted down beside me. “Are you hurt?” she asked.

I sat there, squeezed between the fallen cabinet and the one behind my back. As near as I could tell, nothing was damaged beyond my dignity. I was just stuck and couldn’t get out until someone pulled the rack off me. I shook my head, no.

She smiled a wide, toothy grin. “Well, good. Because Brad himself is going to want to publicly kill you when he finds out what you just did here.”

“What happened to Roger and Shelia and Nidal? They were right here?”

“Oh they left. No big deal, they’re in here all the time. They bring us coffee and sweets to help us make it through the night. They’re great people.”

“Well, then who are you?”

“Oh, I’m part of the third shift data centre crew. You know, data centre operations. We do adds, moves, changes … run some jobs … things like that. That’s why Nidal and his gang are great to have around. On nights like tonight when we are really busy, they’re always happy to give us a hand and do theirs themselves. It helps us meet our schedules and keeps Brad happy.”

My phone went off with a long cascade of severity one outages. Top on the list was Mountain Top. Within seconds, messages from Ramesh, Jessica and Jason popped up on my phone.

A moment later my phone rang, and without thinking, I answered it. It was Ramesh. I wasn’t really surprised. Who else would call me at this hour?

“Chris, where are you?”

“I’m in the data centre.”

“What are you doing there? Don’t you know there is an outage? Mountain Top has gone down. Why aren’t you on your way to the War Room to manage the incident?”

I was please Ramesh got the terminology right; although I had the sense this one would definitely merit examination as a problem.

“I’m just … sitting here getting some assistance from the data centre crew,” I said, peering through the gap between the rack pinning me down and the floor, watching feet gathering around the rack and hands trying to free me.

Ramesh got louder and clearly more frustrated. “Well there is nothing the data centre operators can do for you right now. I want you down in the War Room as quickly as possible.”

“I don’t know about that. Right now, their help is pretty essential. But I’ll be there as quick as I can,” I said, wondering if there were any way I could avoid telling Ramesh about any of this.

Tips that would have helped Chris

If a process is too time-consuming, or requires activities that seem to make no sense to them, people will find ways to work around it. SMEs are goaled on their technical work, not on their ability to fill out forms. Help them be successful and reduce their desire to go around the process. Build each process with their involvement, so it makes sense to them. Help them understand why it is important, and the purpose of everything they are asked to do.

A focus on tools and results comes naturally to most technical specialists. The process, or how they should work, does not. You should take every possible opportunity to educate them why it is important. Since SMEs can often get the support of other SMEs in achieving their objectives if process gets in the way, it is important for you to spread your educational message well beyond the teams directly involved in the areas you are currently improving.