Consider your most important program—one that is strategic, expensive, high profile, and attracts management attention. Suppose you decide to evaluate the success of the program. Through your analysis you find that participants:
• viewed the program as relevant to their work
• acquired new knowledge and skills
• used the knowledge and skills routinely on the job, although they had some difficulty in a few areas
• improved several important work unit measures, including quality and productivity
• achieved a 105 percent return on the investment in the program
• reported an increase in job satisfaction in their work unit.
From an audience reviewing the data, several questions come to mind. Who and what are the sources of the data? What assumptions are made in the analysis? Is the process consistent from one study to another? Is the study credible? What did it cost to produce the study?
From the program owner’s perspective, other questions surface. What would this data mean for your program? What would it mean for your team and you personally? If the above results were negative, what would it mean for the program, your team, and you personally? How should the results be used?
These are the questions faced by hundreds who are beginning their journey into results-based talent development. This book, ROI Basics, 2nd edition, will help you answer these questions and understand the true meaning of return on investment (ROI) in talent development.
The Same, Only Better
The ROI Methodology is a comprehensive evaluation process developed in 1973 by Jack J. Phillips. While the five-level evaluation framework is the basis for categorizing data and based on Raymond Katzell’s four steps of evaluation, the ROI Methodology is the operational process that ensures data collected and categorized within the framework are reliable. More than that, it is a process that enables talent development, and other functions within the organization, to do their job—that is, drive value in the organization. Application of the process is not limited. In fact, the ROI Methodology has reached audiences beyond what we thought it would. Areas in which organizations apply this process include:
• human resources/human capital
• knowledge management/transfer
• work arrangement systems
• change management/culture
• talent management/retention
• compliance/risk management
• organization development/consulting
• project management solutions
• quality/Six Sigma/Lean engineering
• communications/public relations
• public policy/social programs
• safety/health/wellness programs
• healthcare initiatives
• public sector/nonprofits
• faith-based programs.
Some of our most exciting work is with nongovernmental organizations, faith-based organizations, and the First Nation community where we are working with the American Indian Fund to demonstrate the value of tribal colleges.
Never before have we seen such interest in designing programs, processes, initiatives, and institutions to drive results that matter to employees, students, stockholders, taxpayers, and communities. The good news is the process to do this hasn’t changed—it’s the same, only better. Better in terms of the techniques to collect, analyze, report, and optimize data. With greater use, broader adoption, and advances in technology, we continue to learn. We strive to apply that learning by advancing the measurement and evaluation field.
But the fundamentals are the fundamentals—they are foundational and unchanging. They are what makes the ROI Methodology work and why it continues to be the most applied and documented approach to demonstrating impact and ROI for non-capital investments. This book presents those fundamentals. It provides the basics of the ROI Methodology.
While retaining the fundamentals, this second edition improves on the first, which came out in 2005. The question then is why an update? Because the need for ROI is not going away.
Demonstrating the ROI of programs remains critical to how organizations allocate resources to future programs. Senior executives want to know this information. And yet, we keep hearing the same story: Organizations spend millions of dollars on talent development, but the business measures that matter are not improving! That suggests a continuing gap in connecting how a program performs and whether it delivers value.
Here is what we’ve changed in this update:
• We have focused more on alignment, as in connecting programs to the business. This includes identifying payoff needs and specific business measures that need to improve. It also includes using a variety of techniques to identify the most feasible solution given the business need.
• While there is so much to talk about on the technology front, we could have written an entire book just on the application and impact of technology on the adoption of the ROI Methodology. Alas, we did not; although, we do give a shout out to a few technologies that are either fundamental tools or new tools that are sure to take measurement to new levels.
• We have also referenced new applications of the process to talent development, demonstrating its increasing use.
Each chapter provides the basic steps in developing a comprehensive evaluation that includes ROI. Although attempts have been made to address some of the more difficult issues, readers will become most comfortable with the basic techniques. By the end of the book, you will have a basic understanding of the ROI Methodology and be able to select programs for this type of analysis. You will be prepared to develop a strategy to integrate ROI as part of your ongoing talent development process.
Chapter 1, The Basics, provides an overview of ROI—what it means, how it is reported, and when it should be used.
Chapter 2, Plan Your Work, introduces the alignment process, which includes developing program objectives and the evaluation plans that will support your moving forward with an ROI project.
Chapter 3, Collect Data, covers the appropriate data collection procedures and shows how to implement them. This chapter answers the questions: How do you collect data? From whom do you collect data? When do you collect data?
Chapter 4, Isolate Program Impact, addresses one of the most important steps in program evaluation, answering the basic question: How do you know it was your program that improved the measures?
Chapter 5, Calculate ROI, presents the fundamental difference between reporting activity and reporting ROI. It’s in the math. Only by converting impact measures to monetary value and comparing that value to the fully loaded cost of the program can an actual ROI be reported.
Chapter 6, Optimize Results, focuses on communicating results and using black box thinking. Without communication, you can’t accomplish what you set out to accomplish by evaluating the program. Without reflecting on the wins and losses, and nudging people to take action, you cannot improve the program, the system, or the organization. That’s what measurement and evaluation are all about.
Chapter 7, Sustain Momentum, builds on the previous chapter. Anyone can conduct an ROI study, but can you integrate the ROI process into the talent development process so that it is seamless and still effective? This chapter describes how to do that.
The appendix, ROI Forecasting Basics, introduces the concept of forecasting ROI prior to investing in programs, during program implementation, and using Level 3: Application data.
Icons to Guide You
This book has plenty to offer in the way of content that can help you every day. Some icons will alert you to key features of the book.
What’s Inside This Chapter
Each chapter opens with a short introduction that serves as a quick reference to the chapter contents. Use this section to identify the information in the chapter and, if you wish, skip ahead to the material that is most useful to you.
These rules present guiding principles and guidelines to ensure consistent application of the ROI Methodology.
This icon flags sections with greater detail or an explanation about a concept or a principle. Sometimes it is also used for a short but productive tangent.
Think About This
These considerations attempt to present the information in a slightly different way to reinforce learning as well as to generate an “aha moment” that may not have occurred earlier in the chapter.
Getting It Done
The final section of each chapter supports your ability to take the content of that chapter and apply it to your situation. The focus of this section is mostly on job aids and tools for understanding the content. Sometimes it contains a list of questions for you to ponder, sometimes it is a self-assessment tool, and sometimes it lists action steps you can take to improve your skills and help increase the chances for participant success.
Who Should Read This Book?
The book targets beginners who have been challenged to implement a comprehensive evaluation process as well as those who are taking a proactive approach to accountability. However, those who are more advanced, but still question key issues, will find value in reading this book. Talent development managers will also benefit; by understanding the basics, managers can better serve as champions for ROI implementation.
What Do We Mean?
Before delving into the material, let’s clarify a few terms so we’re on the same page. Program refers to the initiative being evaluated. This could be a course, a full-scale change initiative, or a learning management system implementation. Talent development refers to training, performance improvement, learning, development, and education. The levels of evaluation refer to the Phillips five-levels evaluation framework. ROI is defined in the true sense of the acronym—earnings divided by investment or net benefits divided by costs.
We hope this book will help you as you move forward with ROI. Best of luck to all of you who do!
A book is never developed by the authors alone. It begins with the publisher’s willingness to take on the book project and then working with the authors to frame the content in such a way that it will be useful to the target audience. So, our first thanks go to Ann Parker, senior community manager at ATD, for asking us to update the book. We also thank the ATD editorial team for their work in producing the book. ATD has been a long-time partner of ours and we appreciate the opportunity they gave us to lead the measurement and evaluation content for ATD. We believe in the methodology presented here and have observed its successful application in organizations throughout the world. We appreciate ATD’s recognition of its importance.
A huge thanks goes to Hope Nicholas, ROI Institute’s director of publications. Hope jumped on this book and would not let it (or us) out of her line of sight until it was completed. She always comes through to help us develop quality publications. We would also like to thank our entire team at the ROI Institute who make things happen while we work on research and publications. As other authors will attest, developing a book requires deep work—meaning, we get lost in our work and sometimes come up only when hunger strikes. So many thanks to Ann, Kylie, Sherri, Melissa, Becky, Andy, Brady, and Tim for all you do when we’re away!
We’d also like to thank our many workshop participants and clients. Without you, we would have few stories to tell and limited ways in which we could tell them. We appreciate your candor and help in addressing the many issues faced by talent development professionals pursuing this challenging topic.
From Patti to Jack: As always, Jack, you support and encourage me to be my best. I do this by trying to keep up with you! You are my rock, my friend, my love. Thank you!
From Jack to Patti: Much of the success of our work belongs to Patti. She is an outstanding researcher, consultant, teacher, speaker, and author. Her vast knowledge and experience shine through this book. Thank you for the amazing contributions you make each and every day.
All roads lead to ROI!
Patti and Jack Phillips