Appendix: ROI Forecasting Basics – ROI Basics, 2nd Edition

Appendix

ROI Forecasting Basics

Although beyond the scope of this book, it is important to introduce the basics of forecasting. There are a variety of forecasting techniques available. The most common are of pre-program forecasts, pilot programs, and Level 1 forecasts.

Pre-Program Forecasts

Pre-program forecasts are ideal when you are deciding between two programs designed to solve the same problem. They also serve well when considering one very expensive program or deciding between one or more delivery mechanisms. Whatever your need for pre-program forecasting, the process is similar to post-program ROI evaluation.

Noted

When conducting a pre-program forecast, the step of isolating the effects of the program is omitted. It is assumed that the estimated results are referring to the influence on the program under evaluation.

Figure A-1 shows the basic forecast model. An estimate of the change in results data expected to be influenced by the program is the first step in the process. From there data conversion, cost estimates, and the calculation are the same as in post-program analysis. The anticipated intangibles are speculative in forecasting, but they can be indicators of which measures may be influenced beyond those included in the ROI calculation.

Figure A-1. Basic ROI Forecasting Model

There are 10 steps to develop a pre-program ROI forecast:

1.  Develop Levels 3 and 4 objectives with as many specifics as possible.

2.  Estimate or forecast the monthly improvement in the business impact data (ΔP).

3.  Convert the business impact data to monetary values (V) using one or more of the methods described in chapter 5.

4.  Develop the estimated annual impact (ΔI) in monetary terms by multiplying the monthly improvement by the value by 12: ΔI = ΔP × V × 12.

5.  Factor additional years into the analysis if a program will have a significant useful life beyond the first year.

6.  Estimate the fully loaded cost of the program (C), using the cost summary profile shown in chapter 5.

7.  Calculate the forecasted ROI using the total projected benefits and the estimated cost in the standard ROI formula:

8.  Use sensitivity analysis to develop several ROI values with different levels of potential improvements.

9.  Identify potential intangible benefits by obtaining input from those most knowledgeable of the situation.

10.  Communicate the ROI projection and anticipated intangibles with care and caution. Remember: Although you have based the forecast on several clearly defined assumptions, there is still room for error.

Pilot Program

A more accurate forecast of program success is through a small-scale pilot, and then developing ROI based on post-program data. There are five steps to this approach:

1.  As in the pre-program forecast, develop Levels 3 and 4 objectives.

2.  Initiate the program on a small scale without all the bells and whistles. This keeps the cost low without sacrificing the fundamentals of the program.

3.  Fully implement the program with one or more of the typical groups of individuals who can benefit from it.

4.  Calculate the ROI using the ROI Methodology for post-program analysis.

5.  Decide whether to implement the program throughout the organization based on the results of the pilot program.

Using a pilot post-program evaluation as your ROI forecast will allow you to report the actual story of program success for the pilot group, showing results at all five levels of evaluation, including intangible benefits.

Level 1 Forecasting

A simple approach to forecasting ROI for a new program is to add a few questions to the standard Level 1 evaluation questionnaire. As in the case of pre-program forecast, the data are not as credible as in an actual post-program evaluation; however, a Level 1 evaluation at a minimum relies on data from participants who have actually attended the program.

Table A-1 presents a brief series of questions that can develop a forecast ROI at the end of a program. Using this series of questions, participants detail how they plan to use what they have learned and the results that they expect to achieve. They are asked to convert their anticipated accomplishments into an annual monetary value and show the basis for developing the values; they moderate their response with a confidence estimate to make the data more credible while allowing participants to reflect on their uncertainty with the process. Several adjustments are made to the data to develop the total anticipated monetary benefits. The projected costs are developed to compare with the monetary benefits for an ROI calculation. Though not as reliable as actual data, this process provides some indication of potential program success.

Table A-1. Questions for Forecasting ROI at Level 1

1.  As a result of this program, what specific actions will you attempt as you apply what you have learned?

2.  Indicate what specific measures, outcomes, or projects will change as a result of your action.

3.  As a result of these anticipated changes, estimate (in monetary values) the benefits to your organization over a period of one year. $_______________________

4.  What is the basis of this estimate?

5.  What confidence, expressed as a percentage, can you put in your estimate? _______% (0% = no confidence; 100% = complete certainty)

Additional Approaches to Forecasting

Other approaches to forecasting include the use of Level 2 test data. A reliable test, reflecting the content of talent development programs, is validated against impact measures. With a statistically significant relationship between test scores and improvement in impact measures, test scores should relate to improved performance. The performance can be converted to monetary value and the test scores can then be used to estimate the monetary impact from the program. When compared to projected costs, the ROI is forecasted.

Another approach is forecasting ROI at Level 3, which places monetary value on competencies. A very basic approach to forecasting ROI using improvement with competencies is to:

1.  Identify the competencies.

2.  Determine the percentage of the skills applied on the job.

3.  Determine the monetary value of the competencies using the salary and benefits of participants.

4.  Determine the increase in skill level.

5.  Calculate the monetary benefits of the improvement.

6.  Compare the monetary benefits to the cost of the program.

Table A-2 presents a basic example of forecasting ROI using Level 3 data.

Table A-2. Example of Forecasting ROI at Level 3

Ten supervisors attend a four-day learning program

1. Identify competencies: Supervisor skills

2. Determine percentage of skills used on the job: 80% (average of group)

3. Determine the monetary value of the competencies using salary and benefits of participants: $40,000 per participant

Multiply percentage of skills used on the job by the value of the job: $50,000 × 80% = $40,000

Calculate the dollar value of the competencies for the group: $40,000 × 10 = $400,000

4. Determine increase in skill level: 10% increase (average of group)

5. Calculate the monetary benefits of the improvement: $40,000

Multiply the dollar value of the competencies by the improvement in skill level: $400,000 × 10% = $40,000

6. Compare the monetary benefits to the cost of the program: The ROI is 166% and the cost of the program is $15,000

A more comprehensive approach to forecasting ROI at Level 3 is the use of utility analysis. Utility analysis should be considered when it is important to provide monetary value to behavior change.

Forecasting is an excellent tool when an actual ROI study is not feasible. A word of caution, however: if you forecast, forecast frequently. It needs to be pursued regularly to build experience and a history of use. Also, it is always helpful to conduct an actual ROI study following a forecast and compare the results to develop better skills for the forecasting process.

Noted

Forecasting ROI and the use of predictive analytics is becoming much more popular than in the past. Be forewarned: Don’t rely on forecasting alone. While forecasting and predictive analytics are useful, they result in mere estimates of what could be. The real meaning is in what actually occurs—hence, the need for post-program evaluation.