Chapter 12: The Signal Amongst the Noise – Integrated Measurement- KPIs and Metrics


Trey had been at the company only about a year longer than I had. He’d come to the company straight out of school, and he’d flown through school. Word was that he finished a double BS in Computer Science and Data Mining in two years, and an MS in Applied Information Economics in two more. While he didn’t have an extensive work history as an employee, as a graduate student, he’d consulted to a number of marketing firms, advising them on how to deal with big data – the exabytes of information they’d collected, but couldn’t manipulate into useful information without choking. He definitely wasn’t dumb or ignorant, but he sounded like the kind of guy who wears a pocket protector in his pajamas – dull, boring and one dimensional.

His team’s job was to receive the data from Ahermet’s team, analyze it, and produce the reports which were distributed within the company.

“Trey may be the face of the KPI process to the company,” said Art. “But his team is simply the final step in a lengthy chain of activities. Remember,the more handoffs in a process, the greater the chance it will fail to work properly. It’s not appropriate to discuss it with him today, but in the back of your mind think about what we could do to consolidate some of those process steps. We can debrief after the meeting. I’ll expect your assessment so we can come to some joint recommendations.”

That sounded like pure Art. Everything was a cool calculation with a bottom line ROI. Last time I’d heard people talking like that at previous companies, entire teams were eventually laid off or fired, and the survivors ordered to fill in the gaps…or else.

As Art and I walked into Trey’s cube, it was clear he wasn’t the geeky introvert I had anticipated.

The walls of his cube had pictures of him racing motocross. He’d crashed his bike the first month he was working here and broken his arm. I knew because right beside the photos, on the wall of his cube, appeared to be the cast the doctors removed from his arm, which he seemed to have had sealed and mounted on a plaque, with a picture of him being loaded into an ambulance, and another of his twisted cycle. Underneath it was the date, and a quote about how things that don’t kill us make us stronger. I almost laughed when I saw a picture of him surfing on vacation with his cast sealed in some kind of plastic cushion he must have devised. Somehow, I couldn’t imagine his doctor approving that.

“I’m Trey,” he said, as we walked in and sat down. “You must be the exterminators we requested. We’ve been having a terrible problem with bugs…bugs in our software.” With a laugh, he added, “Only kidding. Who is Art and who is Chris? Or should we start by asking what is Art”?

I smiled inside. Trey may have been smart and effusive – still impressed by his own cleverness, but he hadn’t been around long enough to get molded into a shape consistent with the company’s view of the world. Art was going to hate Trey. I could tell already.

Trey was one of the few people I’d seen in IT that used a tablet. I scanned his office and realized there were no laptops or desktops in his office. In most of the places I’d worked before, your status and bragging rights among the technical teams was based on how many different computers you were able to weasel approval for out of accounting. Here was someone whose work was totally computerized, and could reasonably justify several computers. Why was he really eschewing that computing muscle in favor of a vastly, less powerful tablet?

As Trey tapped the screen of his tablet, I asked, “Are you having problems with your laptop”?

Trey shook his head. “Everything we do is in the Cloud – all the apps and all the data. Otherwise, my team would need their own data farm to manage it, and what now takes hours to analyze, would take days. I’m only interested in the results.” Trey looked up and grinned. “I don’t need to overcompensate by filling my office with equipment.” He set the tablet down on the desk and spun it around so we could see it.

Grids of analytical charts slowly scrolled upwards. Most were consistently green. When the rare measurement indicated performance expectations had been broken, the application paused and zoomed in on that measure, freezing it mid-screen until we pressed the continue button.

“This is very impressive,” I said.

Trey shrugged his shoulders. “It’s something I wrote because I wasn’t happy with anything coming out of the SMT tool.It transforms the SMT output into something more useful.”

“Customizing the SMT tool is not allowed. There are too many risks to the…,” I said.

“Risks to the company,” said Trey. “Yes, I know. I’ve heard the same message. Let me ask you this. Which would you rather have, the best information available on which to base your decisions, even though it is contrary to company policy, or some data/information which may be inadequate to make a decision on, but compliant with company policy”?

“Over the long haul, the analysis says although your decisions may be based on weaker information, not customizing is better for the company,” I said.

Trey snickered. “Try talking about that to senior leaders whose compensation is 80% based on the results for the quarter, rather than performance five years out.”

“You didn’t customize SMT, did you”? said Art.

“Smart man,” said Trey. “Never touched it. In fact, we don’t even use its canned reports. We get a raw data feed directly from Ahermet’s team, that is the same one the SMT report writer uses. The only difference is we do a better job of making it useful to our customers. Since we don’t change SMT, and we use the same data…there is no customization. We just know a better way to analyze it.”

Trey spun the tablet back so it faced him. “Of course this is only one view of the information. We have a wide variety of searches that can be conducted.” He paused and looked up at Art. “And for the traditionalists among us, we also provide more familiar printed documents. So what?They show the same information. They’re just not as green, and take an extra day to receive.”

“In all honesty,” he said. “It doesn’t really matter to us what we are analyzing. It’s all data in the Cloud to us.”

“It’s not just any data,” said Art. “I can understand and appreciate how the analysis techniques may not vary much. Leadership is making decisions based on the content of this information.”

“It’s still all numbers. As long as the feeds get to us in a timely fashion, we will meet the deadlines, and produce what people ask,” he said. “That is what leadership consistently tells us they want.”

“It is more than numbers,” I insisted. “You may only see numbers and charts, but they are really shorthand for whether we should be hiring or firing people; starting or stopping new customer services; or potentially even outsourcing or insourcing our service desk activities.”

“You’re misinterpreting me,” said Trey. “We have no vested interests in the results, beyond ensuring what we produce is accurate, and reflects the sensitivities of the consumers of our information.”

“Who are your customers”? I asked.

Trey tapped on his tablet. An enormous list of names began to scroll down the screen. “There are a lot, aren’t there”?

“Too many,” said Art. “Do they all get the same information”?

Trey nodded. “There are three variations; one with mostly numbers for the workers, one with a mix of numbers and charts for the managers, and one that is all pictures for the senior leaders. The content is pretty much the same. It’s all in the presentation.”

Art scribbled more notes. “What about the frequency? Does that stay the same”? asked Art.

“Look, I know where you are going with this. We know that the reports should be crafted for specific audiences, the frequency should match the natural periodic cycle of the underlying processes, and we should cull the distribution lists on a regular basis. Those are some of the first things I had my team do when I got here.”

Trey swiped his tablet clear and leaned across his desk. In a much quieter and subdued voice, he said, “I even had my team put some dummy information in there, to see if anyone was reading them. I went so far as to include the phrase, ‘If you do not call and confirm your desire to continue to receive this report, you will be dropped from further distributions,’ in the middle of the analysis of one KPI.”

Art held his hand up and said, “Wait, let me guess. You got no responses, right”?

Trey nodded. “So I called them on it. I dropped one in every 10names from the distribution list the following cycle. I even dropped 10of the KPIs in the report.”

“Did you get a response”? I asked.

“Response,” said Trey. “I nearly got fired before I finished my first month here. The only people who didn’t scream were those that were no longer here. At least I was allowed to drop them. Apparently, people want to get the report whether they read it or not. If they don’t get it, they assume they are out of the loop on other important things too.”

Art smiled and actually laughed. “Oh yeah, seen it so many times before. What about the reports showing performance expectations always being met? Did you bring that up”?

“That’s a battle I still haven’t won,” said Trey. The frustration was clear in his voice. “I keep getting told that showing anything other than success on every occasion causes concern and doubt in the minds of our customers. Unfortunately, that’s only part of the problem. No one seems to understand the concept of normal variation. It seems like every time I try to share how the KPIs are measured and created, it only creates more anxiety. So I don’t do it anymore.”

“Variation needs to be reduced to zero,” I said. “Everyone knows that. If you eliminate variance, you provide better service. Consistency builds customer confidence.”

“At what cost”? asked Trey. “It’s their resources, so if they want to spend them in pursuit of something they can never achieve, well that is their business. I give the customer what they want; and they want reporting that doesn’t cause them anxiety or confusion.”

“Reducing variation takes resources,” said Art. “The problem is that a certain amount of variance is inherent in any process, if for no other reason than people are involved. Using resources to try to eliminate that is a waste. What you want to look for is abnormal variation…variance beyond a specific range. Your performance expectation should be a range of values, not a single point. As long as the performance is within that range, let it go.”

“That sounds crazy to me,” I said. “How can you possibly know what range to assign? What if it’s one-time versus consistent? It sounds like a great way to let things get out of control.”

Art shook his head, and then looked at Trey. “Don’t they teach statistical process control in business school anymore”?

Trey chuckled. “Not really, and then most people sleep through it.”

Art laughed – louder than I had heard him laugh before. “Thanks a lot, Trey. I think we get the idea of what is happening. If you can send me the current KPI report, Chris and I will leave you to your work. I think we have more than used up our appointed time with you.”

“No problem,” said Trey, as he swiped his fingers across the face of the tablet. “It should be in your e-mail any moment now.”

Art stood and extended his hand across the desk to Trey. “It’s been a real pleasure chatting with you today. It’s really nice to find someone who actually knows what they’re talking about.”

“Anytime you need more information, just let me know,” said Trey.

Art walked out of Trey’s cube and I followed close behind him.

Art led us to the break room, and after grabbing himself a cup of coffee without even bothering to ask if I would like one, he sat down. The room was empty, and I sat down across the table from him.

Referring to his notes, Art said, “Trey’s had some good education, but despite his consulting assignments prior to coming here, he really needs more experience. That’s probably why he couldn’t make people understand the difference between expected common variation and unexpected special variation. Without that understanding, he is going to be locked into over-producing reports, and helping the company continue to waste resources, trying to eliminate something better left alone for now.”

“From my experience,” I said. “I don’t think anyone could convince our leadership of the difference. It’s too conceptual, too theoretical. They don’t want ranges of answers. They want simple ‘Yes-No’ responses. Otherwise it makes their decision-making process too complex.”

“Would you bet your job on that belief”?asked Art

“What do you mean, bet my job”?

“Or better yet, would you bet Trey’s job on that belief”?

I hesitated. “Yes…Yes, I would bet both our jobs on it. I’ve seen enough of how decisions are made here that it doesn’t matter how mathematically perfect the KPIs are, anything beyond a simple answer for a measure will be too much for them. They have too many other issues to deal with, to spend a lot of time analyzing data. They want recommendations. That’s our job.”

“Then what’s the value Trey and his team add”? asked Art. “All they are doing is running Ahermet’s data through an algorithm, and presenting it in a format defined long ago. What have they added as individuals, that an automated machine process could not”?

“That’s not fair. Trey needs an opportunity to educate people and get them to understand. You can’t just toss Trey and his team out on the street. Think about those people. These things take time…”

Art cut me off. “No, time is the one thing we don’t have any to waste. Decisions are being made realtime, right now, that impacts the jobs and futures of everyone at the company. Think about all those people. What if leadership decides to outsource the service desk unnecessarily, based on faulty information passed on by Trey’s team, and the jobs of hundreds of people vanish? Do you think about them? You should.”

Art checked his watch. “I need to go meet with Sergiu. He wants an update on our progress. While I’m gone, you should spend some time collecting your thoughts, and coming up with a set of recommendations for us to present to Sergiu and his peers regarding the KPI process.”

After Art left, I sat alone in the break room. Shortly after Art left, Molly walked in and poured herself a cup of coffee. It wasn’t until she turned to leave that she saw me.

“Oh, hi,” she said. “I almost didn’t notice you there. You were being so quiet. Are you okay”?

“I’m fine. Just finished a session with Trey and Art about analysis and reporting for the KPIs. It was funny. I thought Art was going to hate Trey. Trey seemed like his exact opposite, but Art seemed to like him…until we got here, and then he started talking about him and his team like they were useless.”

“Sounds like Art,” said Molly. “I warned you about him. I’ve seen him in action before. You can never trust him at face value.”

“What does that mean”?I asked.

“I call him Art the secret destroyer,” she said. “You never know what he is thinking. He can be as nice as he needs to be with everyone. But every time I’ve seen him in a company, people lose their jobs.”

“Does it make me different that he treats me like an idiot”?

Molly laughed and shook her head. “Nope. It just means he doesn’t view you as a potential threat, or someone who could derail his plans. So he doesn’t feel he needs to spend any energy being discreetaround you. You get a deeper view into what he is thinking.”

“Well, it’s pretty nasty,” I said.

“I don’t like to say anything bad about anyone unless it is absolutely called for,” she said. “Still, you should be wary of him, even if you’re not a threat to him right now. He will wait for just the right time to humiliate you and your ideas in front of your managers and leaders.”

“So what do I do”?I asked.

“Let’s take a look at the material we’ve collected, and try to figure out which things he will think are the weakest or hardest to explain. That’s what he’ll go after. What he doesn’t know is that we will be ready for him.”

Molly and I began working through the material, identifying areas Art was likely to dispute. Art may have been more than a match for anyone, but with Molly’s help, I was going to become more than a match for him.

Tips that would have helped Chris

It is more than just lines on paper

People want control over how their information is reported and distributed. They view it as exposing their secrets to the world. This is especially true in corporate cultures, where any signs of weakness or stumbles can be fatal; cultures that don’t understand the difference between signal and noise, or common and abnormal variation. This type of culture will make it challenging to produce objective, transparent reporting of KPI achievement, without first educating people about the nature of variation.

Printing reports doesn’t make the content true

It is human nature to assume that no matter how inaccurate or inconsistent the data stream is flowing into an automated system, once it is regurgitated by that system, it is somehow cleansed, correct and accurate. The old phrase “Garbage In – Garbage Out” is just as important and true now as it always was.

Reports are about communicating, not art

Sometimes people confuse making a report’s information easy to understand, with making it look pretty. Eye candy is a distraction, and only disengages people from the information flow. Edward Tufte has written some excellent works on how to find the right balance.