Isolate Program Impact
What’s Inside This Chapter
This step in the ROI Methodology attempts to delineate the direct contribution caused by the talent development program, isolating it from other influences. This chapter covers three critical areas:
• understanding why isolating impact is a key issue
• identifying the methods to do it
• building credibility with the process.
Understanding Why Isolating Impact Is a Key Issue
Isolating the effects of a program on business impact data is one of the most challenging yet necessary steps in the ROI Methodology. When addressed credibly, this step links learning directly to improvement in key performance measures.
Other Factors Always Exist
In almost every situation, multiple factors influence organizational performance. The world does not stand still as employees participate in a talent development program. Many functions in the organization are attempting to improve the same metrics as the talent development function. A situation where no other factors enter into the process would be rare. Important arguments exist that support the need to take this step.
Without Isolating Impact, There Is No Alignment—Evidence Versus Proof
Without taking steps to show the contribution, the alignment between programs and improvement in organizational measures does not exist. There may be evidence that a program might make a difference, but there is no accounting for what other factors may have contributed to the improvement. Proving the connection between programs and performance improvement is what this step in the process is all about—isolating the effects of the program.
Other Factors and Influences Have Protective Owners
The owners of the other processes influencing results are convinced that their processes made the difference. When sales increase, marketing and advertising each believe their efforts are the cause and they present a compelling case to management, stressing their achievements. The IT department also believes that technology made the difference. They, likewise, can present a compelling case. In real situations, other processes, such as performance improvement, reward systems, and job redesign, have protective owners, and those owners often can be very convincing that they made a difference.
The challenge of isolating the effects of the program on impact data is critical and can be done; however, it is not easy for very complex programs, especially when strong-willed owners of other processes are involved. It takes a determination to address this situation every time an ROI study is conducted. Fortunately, a variety of approaches is available.
Think About This
You have conducted a sales training program to improve sales competencies for client relationship managers. This program is designed to increase sales as the managers use the competencies. Three months after the training, sales have increased. However, during the evaluation period, product marketing and promotion increased. Also, prices were lowered in two key product lines and new technologies enabled the sales representatives to secure quotes faster, thus increasing efficiency and boosting sales. All these factors influence sales. From the perspective of the sales training function, the challenge is to determine how much of the sales increase is due to the training. If a method is not implemented to show the contribution and talent development claims full credit for improvement in measures, the talent development staff will lose credibility.
Without Isolating Impact, the Study Is Not Valid
Without addressing this issue, your evaluation study is not valid because there are almost always other factors in the mix and the direct connection to learning is not apparent.
Above all, without first isolating the impact, there are two things that you should not do:
1. Take all the credit for the improvement without tackling the issue.
2. Do nothing, attempting to ignore the issue altogether.
Neither of these help the talent development team connect their efforts to the business.
Myths About Isolating the Effects of the Program
Several myths about isolating the effects of the program permeate talent development functions and create concerns, confusion, and frustration with this process. Some researchers, talent development professionals, and consultants inflame this matter by suggesting that isolating the effects is not necessary. Here are the most common myths:
• Talent development is complementary with other processes; therefore, you should not attempt to isolate its effects on performance improvement. True, talent development programs are complementary to other factors, all of which drive results. But if a project sponsor needs to understand the relative contribution of talent development, you must tackle isolating the impact. If accomplished properly, it will show how all the complementary factors are working together to drive the improvements.
• Other functions in the organization do not isolate the effects. While some functions do not grapple with this issue because they try to make a convincing case that the improvement is related to their own processes, others do isolate their effects. You need a credible approach to address this issue. Notice the next time you complete a customer survey after you make a purchase or open a new account—do they ask you why you made the purchase? They are trying to isolate the results of multiple variables.
• If you cannot use a comparison group analysis (a research-based control group), then you should not attempt this step. Although a comparison group analysis is the most credible approach, using one is not feasible in all situations. Consequently, other methods must be used to isolate the effects. The problem does not go away just because you cannot use your desired or favorite method. The challenge is to find other methods that are effective and will work anytime, even if they are not as credible as the comparison group method.
• The stakeholders will instinctively understand the linkage; therefore, you do not need to attempt to isolate the effects of learning on impact measures. Unfortunately, stakeholders see and understand what is presented to them. If they do not see the linkage, it is likely they will assume there is none. In fact, in 2009, ATD and the ROI Institute conducted a study to determine what CEOs thought about their investment in talent development. Of the findings, one was that 74 percent of CEOs wanted to see a direct connection between talent development and improvement in impact measures, yet only 4 percent were seeing it (Phillips and Phillips 2009). Another study by Harvard Business Review Analytic Services (2017) reported that only 24 percent of C-suite executives are provided data that connect people metrics to business metrics. Without the proper information, stakeholders will struggle to understand the linkage, particularly when others are claiming full credit for the improvement.
• Estimates of improvement provide no value. The last-resort scenario is to tackle isolating the impact by using estimates from the individuals who understand the process the most. Although this should be your last choice, it may provide value and be a credible process, particularly when the estimates are adjusted for the error of the estimate. Estimates are used routinely in other functions.
• Ignore the issue; maybe they won’t think about it. Unfortunately, audiences are becoming more sophisticated regarding isolating impact, and they are aware of multiple influences. If no attempt is made to isolate the effects of learning, the audience will assume that the other factors have had a tremendous effect, and maybe all the effect. Thus, credibility deteriorates.
These myths underscore the tremendous importance of addressing this issue. If you do not tackle this issue, others will—leaving talent development with less than their desired budgets, resources, and respect. This is not to suggest that talent development does not work in harmony with other processes. All groups should be working together to move organizations in the right direction. However, when funding is provided to different functions in the organization—with different process owners—there is always a struggle to show, and sometimes even to understand, the connection between what they do and the results.
Applying the Techniques
Before the specific techniques are discussed, it is helpful to review two important principles. First, the chain of impact should be revisited. Although the isolation step can be conducted on Level 3: Application and Implementation data (separating the influence of other factors on the actual behavioral change), it is usually applied to Level 4: Impact data. This is the level where the concerns are most frequently raised. The amount of impact connected to the program is a key issue. After the impact data have been collected, the next step in the analysis is to isolate the effects of the program. This is the proof that talent development made a difference.
Second, there needs to be an attempt to identify the other factors that have contributed to the improvement in the business results measures. This step recognizes that other factors are almost always present and that the credit for improvement is shared with other functions in the organization. Just taking this step is likely to gain respect from the management team.
Basic Rule 5
Use at least one method to isolate the effects of a project.
Several potential sources can help identify these influencing factors. The sponsors of the project may be able to identify the factors. Subject matter experts, process owners, and those who are most familiar with the situation may be able to indicate what has changed to influence the results. In many situations, participants know what other factors have influenced their performance. After all, it is their direct performance that is being measured and monitored.
By taking stock of this issue, all factors that contributed to improvement are revealed, indicating the seriousness of the issue and underscoring how difficult it is going to be to isolate the effects of the program.
The following are four techniques most frequently used to isolate the effects of talent development programs: control group arrangement, trend line analysis, mathematical modeling, and expert estimation.
Control Group Arrangement
The most accurate and credible approach to isolating the effects of talent development programs is control group arrangement or experimental design. This approach involves the use of an experimental group that attends the program and a control group that does not. The composition of both groups should be as similar as possible, and, if feasible, the selection of participants for each group should be on a random basis. When this is possible and both groups are subjected to the same environmental influences, the differences in the performance of the two groups can be attributed to the training program.
There are different approaches to experimental design. One is the classic design where pre-program measures are collected prior to the program and then the change in performance for each group is compared. There is also post-program-only design (Figure 4-1). In this case, the control group and experimental group are either performing at the exact same time or there is not a pre-program measurement. Measurements are taken after the program is implemented. The difference in the post-program performance of the two groups shows the amount of improvement that is directly related to the training program.
For the control group arrangement to be used, six conditions must be met:
1. One or two measures need to be identified that represent the outcome of the program. This is what is used to compare the control and experimental groups.
2. In addition to the talent development program, the factors that influence the outcome measures can be identified and the two groups are matched accordingly.
3. There are enough participants available from which to select the two groups.
4. The program training can be withheld from the control group without any operational problems.
5. The same environmental influences affect both groups during the experiment, with the talent development program being the only difference.
6. The experimental group is protected from contamination. In other words, the system for program implementation is such that members of the control group do not pick up and apply elements of the program.
If these conditions are met, there is a possibility for control group arrangement.
Retail Merchandise Company (RMC) is a national chain of 420 stores. The executives at RMC were concerned about the slow sales growth and were experimenting with several programs to boost sales. One of their concerns focused on the interaction with customers. Sales associates were not actively involved in the sales process, usually waiting for a customer to make a purchasing decision and then proceeding with processing the sale. Several store managers had analyzed the situation to determine if more communication with the customer would boost sales. The analysis revealed that simple techniques to probe and guide the customer to a purchase should boost sales in each store.
The senior executives asked the talent development staff to experiment with a customer interactive skills program for a small group of sales associates. The training staff would prefer a program produced by an external supplier to avoid the cost of development, particularly if the program was not effective. The specific charge from the management team was to implement the program in three stores, monitor the results, and make recommendations.
The talent development staff selected an interactive selling skills program, which makes significant use of skill practices. The program includes two days of training in which participants have an opportunity to practice each of the skills with a fellow classmate, followed by three weeks of on-the-job application. Then, there’s a final day of training that includes a discussion of problems, issues, barriers, and concerns about using the skills. Additional practice and fine-tuning of skills also take place in the final one-day session. At RMC, this program was tried in the electronics area of three stores, with 16 people trained in each store.
One of the most important parts of this evaluation is isolating the effects of the training program. This is a critical issue in the planning stage. The key question is, “When sales data are collected three months after the program is implemented, how much of the increase in sales, if any, is directly related to the program?” Although the improvement in sales may be linked to the talent development program, other nontraining factors contribute to improvement. Though the cause-and-effect relationship between training and performance improvement can be very confusing and difficult to prove, it can be accomplished with an acceptable degree of accuracy. In the planning process, the challenge is to develop one or more specific strategies to isolate the effects of training and include it on the ROI analysis plan.
In this case study, the issue was relatively easy to address. Senior executives gave the talent development staff the freedom to select any stores for implementation of the pilot program. The performance of the three stores selected for the program was compared with the performance of three other stores that were identical in every way possible. This approach represents the most accurate way to isolate the effects of a program. Although other strategies, such as trend line analysis and estimation, would have also been feasible, the control group analysis was selected because of the appropriateness of the situation and the credibility of the analysis. The challenge in using a control versus experimental group is to appropriately select both sets of stores.
Think About This
You have been tasked with developing the criteria to match the control and experimental groups in this case study. What are your criteria for matching the two groups?
It was important for those stores to be as identical as possible, so the talent development staff developed several criteria that could influence sales. This list became quite extensive and included market data, store-level data, management and leadership data, and individual differences. In a conference call with regional managers, this list was pared down to the four most likely influences. The executives selected those influences that would account for at least 80 percent of the differences in weekly store sales per associate. These criteria were as follows:
• store size, with the larger stores commanding a higher performance level
• store location, using a market variable of median household income in the area where customers live
• customer traffic levels, which measures the flow of traffic through the store; this measure was originally developed for security purposes
• previous store performance, a good predictor of future performance; the talent development staff collected six months of data on weekly sales per associate to identify the two groups.
While other factors could have had an influence on sales, there was up-front agreement that these four criteria would be used to select three stores for the pilot program and match them with three other stores. As a fallback position, in case the control group arrangement did not work, participant estimates were planned.
Disadvantages and Advantages
While the gold standard in demonstrating cause-and-effect relationships, the control group arrangement does have some inherent problems that may make it difficult to apply. First is that the process is inappropriate for many situations. For some types of talent development programs, it is not proper to withhold a program from one group while engaging another. This is particularly important for critical skills that are needed immediately on the job. For example, in entry-level training, employees need basic skills to perform their jobs. It would be improper to withhold training from a group of new employees just so they can be compared with a group that receives the training. Although this would reveal the effect of initial training, it would further handicap those individuals who are struggling to learn necessary skills and trying to cope with the job situation. Situations like the previous case study describing the use of control group to demonstrate improved store sales are feasible. The training provided was not necessarily essential to the job, and the organization was not completely convinced that it would add value to actual sales.
This barrier keeps many control groups from being implemented. Management is not willing to withhold a program in one area to see how it works in another. However, in practice, there are many opportunities for a naturally occurring control group to develop in situations where programs are implemented throughout an organization. If it will take several months for everyone to participate in a program, there may be enough time for a parallel comparison between the initial group and the last group. These naturally occurring control groups often exist in major talent development program implementations.
A second problem is that the control groups must be addressed early enough to influence the implementation schedule so that similar groups can be used in the comparison. Dozens of factors can affect employee performance, some of them individual and others contextual. To tackle the issue on a practical basis, it is best to select three to five variables that will have the greatest influence on performance.
A third problem with the control group arrangement is contamination, which can occur when participants in the program influence others in the control group. Sometimes the reverse situation occurs when members of the control group model the behavior from the trained group.
In either case, the experiment becomes contaminated because the influence of the program filters to the control group. This can be minimized by ensuring that control groups and experimental groups are at different locations, have different shifts, or are on different floors in the same building. When this is not possible, it is sometimes helpful to explain to both groups that one group will receive training now and another will receive training at a later date. Also, it may be helpful to appeal to the sense of responsibility of those being trained and ask them not to share the information with others.
A challenge is when the control group outperforms the experimental group. In some cases, the program was, in fact, a poor solution to the opportunity. But more times than not, when the control group outperforms the experimental design, there is a problem with the research design. Therefore, it is important to have an alternative approach readily available to determine how much improvement is due the program.
Fourth, and closely related to the previous problem, is the issue of time. The longer a control group and experimental group comparison goes on, the greater the likelihood that other influences will affect the results. More variables will enter into the situation, contaminating the results. On the other end of the scale, there must be enough time so that a clear pattern can emerge between the two groups. Thus, the timing for control group arrangement must strike a delicate balance of waiting long enough for their performance differences to show but not so long that the results become seriously contaminated.
A fifth problem occurs when the different groups function under different environmental influences because they may be in different locations. Sometimes the selection of the groups can help prevent this problem from occurring. Also, using more groups than necessary and discarding those with some environmental differences is another tactic.
A sixth problem with using control groups is that it may appear to be too research oriented for the organization. For example, management may not want to take the time to experiment before proceeding with a program, or they may not want to withhold a program from a group just to measure the impact of an experimental program. Because of this concern, some professionals do not entertain the idea of using control groups. When the process is used, however, some organizations conduct it with pilot participants as the experimental group and nonparticipants as the control group. Under this arrangement, the control group is not informed of their control group status.
The primary advantage of using control versus experimental groups is that it is the gold standard in demonstrating cause and effect. This level of credibility is important when reporting results of a major talent development initiative. When the experimental and control groups are evenly matched and the program is the only other factor, it’s difficult for someone to push back on results. In today’s era of agility and analytics, experimentation is becoming more acceptable than in the past. Senior leaders recognize the need to pivot quickly if a program is not working; so, rather than invest in a complete rollout of an initiative, testing with a smaller group today can avoid a bad investment decision in the future.
Trend Line Analysis
Another technique used to isolate the impact of talent development programs is the forecasting and trend line analysis process. This approach has credibility if it is feasible and, when appropriate, is an alternative to control group arrangement.
A trend line is drawn using pre-program performance as a base and extending the trend into the future. After the program is conducted, actual performance is compared to the projected value, the trend line. Any improvement of performance over what the trend line predicted can then be reasonably attributed to the program. For this to work, four conditions must exist:
1. Pre-program data are available. These represent the impact data—the proposed outcome of the program. While the number of data points required to make a trend depends on the data, six data points would be the minimum.
2. Pre-program data should be stable, not erratic.
3. The trend that has developed prior to the program is expected to continue if the program is not implemented to alter it.
4. No other new variables enter the process after the program is conducted. The key word is “new,” realizing that the trend has been established because of the variables already in place, and no additional variables have entered the process beyond the talent development program.
In a warehouse where documents are shipped to fill consumer orders, shipment productivity is routinely monitored. For one particular team, the shipment productivity is well below where the organization desires it to be. The ideal productivity level is 100 percent, reflecting that the actual shipments equal the scheduled shipments.
Figure 4-2 shows the data before and after the team training program. As shown in the figure, there was an upward trend on the data prior to conducting the training. Although the program apparently had a dramatic effect on shipment productivity, the trend line shows that improvement would have continued anyway, based on the trend that had been previously established. It is tempting to measure the improvement by comparing the average six months of shipments prior to the program (87.3 percent) to the actual average of six months after the program (94.4 percent), yielding a 7.1 percentage point difference. However, a more accurate comparison is the six-month actual average after the program (94.4 percent) compared with the trend line (92.3 percent); the difference is 2.1 percentage points. This comparison accounts for the trend (and all factors influencing the trend), as well as the program
If the variance of the data is high, the stability of the trend line becomes an issue. If this is an extremely critical issue and the stability cannot be assessed from a direct plot of the data, more detailed statistical analyses can be used to determine if the data are stable enough to make the projection. The trend line can be projected with a simple formula available in many calculators and software packages.
Disadvantages and Advantages
The primary disadvantage of the trend-line approach is that it is not always accurate. The use of this approach assumes that the events that influenced the performance variable prior to the program are still in place after the program, except for the implementation of the training program. Also, it assumes that no new influences entered the situation at the time the training was conducted. This is seldom the case.
The primary advantage of this approach is that it is simple and inexpensive. If historical data are available, a trend line can quickly be drawn and differences estimated. Although not exact, it does provide a quick assessment of a talent development program’s potential results.
A more analytical approach to isolating program effects is the use of mathematical modeling. Correlation does not equal causation; this has been proven time and again. Yet, scientists continue to demonstrate approaches in which, under certain circumstances, correlational analysis indicates causal outcomes (Mooij et al. 2016). Through the development of models using robust statistical analysis, an evaluator can demonstrate a relatively reliable connection between performance variables. This approach represents a mathematical interpretation of the trend-line analysis when multiple variables enter the situation at the time of a program.
The basic premise is that the actual performance of a measure, related to talent development, is compared to movement of the measures related to other factors. A critical path is formed that can lead one to ascertain that some amount of improvement in the measure can be explained by a talent development program. This form of analysis is becoming more popular given the amount of data and technology now available. However, it is still not as accessible to evaluators of talent development as in other areas.
A basic example of how this type of analysis can be employed is in a retail setting where two investments were being made to increase sales: advertising and training. The marketing and advertising team tracked investment in advertising and sales over time. Using the method of least squares, they found that there was a mathematical relationship between advertising and sales: Y = 140 + 40X, where Y represented the daily sales per employee and X represented the investment in advertising divided by 1,000. Prior to the program the average daily sales, using a one-month average, was $1,100. The investment in advertising was $24,000. In formula form: $1,100 = 140 + 40(24).
Six months after the program, average sales on a monthly basis was $1,500. Investment in advertising was $30,000. However, there had been a training program during that six months. While the senior executive and talent development team discussed other factors, they agreed the only other “significant” factor that could have influenced sales was the training program. To account for the increase, the first step was to solve for the contribution of advertising using the mathematical formula: Y = 140 + 40(30). The output showed that average sales due to advertising was $1,340. The difference between the $1,500 and $1,340 was $160. This difference was attributed to the training program.
Disadvantages and Advantages
The major disadvantage with mathematical modeling occurs when several variables enter the process. The complexity of analytics multiplies, and the use of sophisticated statistical packages for multiple-variable analyses is necessary. Even then, a good fit of the data to the model may not be possible. Unfortunately, some organizations have not developed mathematical relationships for output variables as a function of one or more inputs. Without them, this approach is difficult to use.
The primary advantage of this type of analysis is that more organizations are investing in development of analytics capability. With this capability and the use of robust analysis, it is becoming more accessible to make accurate predictions of organization performance measures with and without talent development programs.
An easily implemented method to isolate the effect of learning is to obtain information directly from experts who understand the business performance measures. The experts could be any number of individuals, including participants, supervisors, managers, sponsors, subject matter experts, process owners, external experts, and customers. For many programs, the participants are the experts. After all, the measure is reflecting their individual performance. They may know more about the relationships between the different factors, including learning, than any other individual.
Because of the importance of estimations from program participants, much of the discussion in this section relates to how to collect this information directly from participants. The same methods would be used to collect data from others. The effectiveness of the approach rests on the assumption that participants are capable of determining how much of a performance improvement is related to the training program. Because their actions have produced the improvement, participants may have very accurate input on the issue. Although an estimate, this value will typically have credibility with management because participants are at the center of the change or improvement.
When using this technique, four assumptions are made:
1. A talent development program has been conducted with a variety of different activities, exercises, and learning opportunities, all focused on improving performance.
2. Business measures have been identified prior to the program and have been monitored following the program. Data monitoring has revealed an improvement in the business measure. (The process starts with this step.)
3. There is a need to link the talent development program to the specific amount of performance improvement and develop the monetary effect of the improvement. This information forms the basis for calculating the actual ROI.
4. The participants are capable of providing knowledgeable input on the cause-and-effect relationship between the different factors, including learning and the output measure.
With these assumptions, the participants can pinpoint the results linked to the program and provide data necessary to develop the ROI. This can be accomplished by using a focus group or a questionnaire.
Focus Group Approach
The focus group approach works extremely well for this challenge if the group size is relatively small—in the eight to 12 range. If much larger, the groups should be divided into multiple groups. Focus groups provide the opportunity for members to share information equally, avoiding domination by any one individual. The process taps the input, creativity, and reactions of the entire group.
The focus group session should take about one hour (slightly more if there are multiple factors affecting the results or there are multiple business measures). The facilitator should be neutral to the process (that is, the same individual conducting the program should not conduct this focus group).
The task is to link a talent development program to business performance. The group is presented with the improvement, and they provide input on how much of the improvement is due to the program. Twelve steps are recommended to arrive at the most credible value for learning impact:
1. Explain the task. The first step is to describe the task to members of the focus group. Participants should be clear that there has been improvement in performance. While many factors could have contributed to the performance, the task of this group is to determine how much of the improvement is related to the specific program.
2. Discuss the rules. Each participant should be encouraged to provide input, limiting comments to two minutes per person for any specific issue. Comments are confidential and will not be linked to a specific individual.
3. Explain the importance of the process. The participant’s role in the process is critical. Because it is their new actions, behaviors, or processes that have led to performance improvement in measures related to their work, they are in the best position to indicate what has caused this improvement; they are the experts in this process. Without quality input, the contribution of the program (or any other processes) may never be known.
4. Select the first measure and show the improvement. Using actual data, show the level of performance prior to and following the program; in essence, the change in business results is reported. If the participants have individual data, the individual improvements should be used.
5. Identify the different factors that have contributed to the performance. Using input from experts and process owners—others who are knowledgeable about the improvements—identify the factors that have influenced the improvement (for example, the volume of work has changed, a new system has been implemented, or technology has been enhanced). If these are known, they are listed as the factors that may have contributed to the performance improvement.
6. Identify other factors that have contributed to the performance. In some situations, only the participants know other influencing factors, and those factors should surface at this time.
7. Discuss the linkage. Taking each factor one at a time, the participants individually describe the linkage between that factor and the results using a time limit of two minutes. For example, for the program influence, the participants would describe how the program has driven the actual improvement by providing examples, anecdotes, and other supporting evidence. Participants may require some prompting to provide comments. If they cannot provide dialogue on this issue, there’s a good chance that that factor had no influence.
8. Repeat the process for each factor. Each factor that could have influenced performance in the measure is explored until all the participants have discussed the linkage between all the factors and the business performance improvement. After this linkage has been discussed, the participants should have a clear understanding of the cause-and-effect relationship between the various factors and the business improvement.
9. Allocate the improvement. Participants are asked to allocate the percentage of improvement to each of the factors discussed. Participants are provided a pie chart that represents a total amount of improvement for the measure in question and are asked to carve up the pie, allocating the percentages to different improvements with a total of 100 percent. Some participants may feel uncertain with this process but should be encouraged to complete it using their best estimate. Uncertainty will be addressed later in the meeting.
10. Provide a confidence estimate. The participants are then asked to review the allocation percentages and, for each one, estimate their level of confidence in the allocation estimate. Using a scale of 0 to 100 percent, where 0 percent represents no confidence and 100 percent is complete certainty, participants express their level of certainty with their estimates in the previous step. A participant may be more comfortable with some factors than others, so the confidence estimate may vary. This confidence estimate serves as a vehicle to adjust results.
11. Ask the participants to multiply the two percentages. For example, if an individual has allocated 35 percent of the improvement to learning and is 80 percent confident, they would multiply 35 percent by 80 percent, which is 28 percent. In essence, the participant is suggesting that at least 28 percent of the business improvement is linked to the talent development program. The confidence estimate serves as a conservative discount factor, adjusting for the error of the estimate. The pie charts with the calculations are collected without names, and the calculations are verified. Another option is to collect the pie charts and make the calculations for the participants.
12. Report results. If possible, the average of the adjusted values is developed and communicated to the group. Also, the summary of all the information should be communicated to the participants as soon as possible. Participants who do not provide information are excluded from the analysis.
Table 4-1 illustrates the focus group estimation approach with an example of one participant’s estimates. This participant allocates 50 percent of the improvement to the talent development program. The confidence percentage is a reflection of possible error in the estimate. A 70 percent confidence level equates to a potential error range of ±30 percent (100 percent × 70 percent = 30 percent). The 50 percent allocation to the program represents ±15 percent (50 percent × 30 percent = 15 percent). Thus, the contribution could be 65 percent (50 percent + 15 percent = 65 percent) or 35 percent (50 percent – 15 percent = 35 percent) or somewhere in between.
The participant’s allocation is in the range of 35 to 65 percent. In essence, the confidence estimate frames this error range. To be conservative, the lower side of the range is used (35 percent).
This approach is equivalent to multiplying the factor estimate by the confidence percentage to develop a usable learning factor value of 35 percent. This adjusted percentage is then multiplied by the actual amount of the improvement (post-program minus pre-program value) to isolate the portion attributed to the program. The adjusted improvement is now ready for conversion to monetary values and, ultimately, for use in developing the return on investment.
This approach provides a credible way to isolate the effects of the program when other methods will not work. It is often regarded as the low-cost solution to the problem because it takes only a few focus groups and a small amount of time to arrive at this conclusion. In most of these settings, the actual conversion to monetary value is not conducted by the group but developed in another way. For most data, the monetary value may already exist as a standard, acceptable value. However, if the participants must provide input on the value of the data, it can be approached in the same focus group meeting as another phase of the process in which the participants provide input into the actual monetary value of the unit. To reach an accepted value, the steps are very similar to the steps for isolation.
Basic Rule 6
Adjust estimates of improvement for potential errors of estimation.
Sometimes focus groups are not available or are considered unacceptable for use in data collection. The participants may not be available for a group meeting, or the focus groups may become too expensive. In these situations, it may be helpful to collect similar information via a questionnaire. With this approach, participants address the same issues as those addressed in the focus group, but now on a series of impact questions imbedded in a follow-up questionnaire.
The questionnaire may focus solely on isolating the effects of talent development, as detailed in the previous example, or it may focus on the monetary value derived from the program, with the isolation issue being only a part of the data collected. Using questionnaires is a more versatile approach when it is not certain exactly how participants will provide impact data. In some programs, the precise measures that will be influenced by the program may not be known. This is sometimes the case in programs involving leadership, team building, communications, negotiations, problem solving, innovation, and other types of talent development initiatives. In these situations, it is helpful to obtain information from participants on a series of impact questions, showing how they have used what they have learned and how the work unit has been affected. It is important for participants to know about these questions before they receive the questionnaire; otherwise, they may not respond or may struggle with their responses.
The following is a series of questions that will lead to determining the improvement in impact measures due to the program and the value of that improvement. Questions 8, 9, and 10 are those necessary to isolate program effects on performance improvement.
1. How have you and your job changed as a result of attending this program (knowledge and skills application)?
2. What effects do these changes bring to your work or work unit?
3. How is this effect measured (specific measure)?
4. How much did this measure change after you participated in the program (monthly, weekly, or daily amount)?
5. What is the unit value of the measure?
6. What is the basis for this unit value? Please indicate the assumption made and the specific calculations you performed to arrive at the value.
7. What is the annual value of this change or improvement in the work unit (for the first year)?
8. Recognizing that many other factors influence output results in addition to the program, please identify the other factors that could have contributed to this performance.
9. What percentage of this improvement can be attributed directly to the application of skills and knowledge gained in the program? (0 percent to 100 percent)
10. What confidence do you have in the above estimate and data, expressed as a percentage? (0 percent = no confidence; 100 percent = complete certainty)
11. What other individuals or groups could estimate this percentage or determine the amount?
Perhaps an illustration of this process can reveal its effectiveness and acceptability. In a large global organization, the impact of a leadership program for new managers was being assessed. Because the decision to calculate the impact of the program was made after the program had been conducted, control group arrangement was not feasible as a method to isolate the effects of training. Also, before the program was implemented, no specified impact data were identified as directly linked to the program. Participants may drive one or more of a dozen business performance measures. Consequently, it was not appropriate to use trend line analysis. Participants’ estimates proved to be the most useful way to assess the impact of the program on performance improvement. In a detailed follow-up questionnaire, participants were asked a variety of questions regarding the applications of what was learned from the program. As part of the program, the individuals were asked to develop action plans and implement them, although there was no specific follow-up plan needed.
Although this series of questions is challenging, when set up properly and presented to participants in an appropriate way, it can be very effective for collecting impact data. Table 4-2 shows a sample of the calculations from these questions for this particular program.
Although this is an estimate, the approach has credibility. Four adjustments are effectively used with this method to reflect a conservative approach:
• The individuals who do not respond to the questionnaire or provide usable data on the questionnaire are assumed to have no improvements. This is probably an understatement of results because some individuals will have improvements but not report them on the questionnaire.
• Extreme data and incomplete, unrealistic, and unsupported claims are omitted from the analysis, although they may be included in the intangible benefits.
• Because only annualized values are used, it is assumed that there are no benefits from the program after the first year of implementation. In reality, leadership development should be expected to add value for several years after the program has been conducted.
• The confidence level, expressed as a percentage, is multiplied by the improvement value to reduce the amount of the improvement by the potential error.
Basic Rule 7
If no improvement data are available for a population or from a specific source, it is assumed that little or no improvement has occurred.
When presented to senior management, the results of this study were perceived to be an understatement of the program’s success. The data and the process were considered to be credible and accurate.
Collecting an adequate amount of quality data from the series of impact questions is the critical challenge with this process. Participants must be primed to provide data, and this can be accomplished in six ways.
• Participants should know in advance that they are expected to provide this type of data along with an explanation of why the information is needed and how it will be used.
• Ideally, participants should see a copy of this questionnaire and discuss it while they are involved in the program. If possible, a verbal commitment to provide the data should be obtained at that time.
• Participants could be reminded of the requirement prior to the time to collect data. The reminder should come from others involved in the process—even the immediate manager.
• Participants could be provided with examples of how the questionnaire can be completed, using likely scenarios and types of data.
• The immediate manager could coach participants through the process.
• The immediate manager could review and approve the data.
Basic Rule 8
Avoid use of extreme data items and unsupported claims when calculating ROI.
These steps help keep the data collection process with its chain of impact questions from being a surprise. It will also accomplish three critical tasks:
• The response rate will increase. Because participants commit to provide data during the session, a greater percentage will respond.
• The quantity of data will improve. Participants will understand the chain of impact and understand how data will be used. They will complete more questions.
• The quality of the data is enhanced. With up-front expectations, there is greater understanding of the type of data needed and improved confidence in the data provided. Perhaps subconsciously, participants begin to think through consequences of training and specific result measures.
Basic Rule 9
Use only the first year of annual benefits in ROI analysis of short-term solutions
Disadvantages and Advantages
Participant estimation is a useful technique to isolate the effect of talent development; however, the process has some disadvantages. It is an estimate and, consequently, does not have the accuracy desired by some talent development managers. Also, the input data may be unreliable because some participants are incapable of providing these types of estimates. They might not be aware of exactly which factors contributed to the results or they may be reluctant to provide data. If the questions come as a surprise, the data will be scarce.
Several advantages make this strategy attractive. It is a simple process, easily understood by most participants and by others who review evaluation data. It is inexpensive, takes very little time and analysis, and thus results in an efficient addition to the evaluation process. Estimates originate from a credible source—the individuals who produced the improvement.
The advantages seem to offset the disadvantages. Isolating the effects of talent development will never be precise, but this estimate may be accurate enough for most clients and management groups. The process is appropriate when the participants are managers, supervisors, team leaders, sales associates, engineers, and other professional and technical employees.
This technique is the fallback isolation strategy when other techniques will not work. It is a fallback approach for when the effects of learning must be isolated and no other technique is feasible. Trainers, training managers, learning specialists, and performance improvement specialists are often reluctant to use a technique that lacks absolute precision. However, the primary audience for the data (the sponsor or senior manager) will readily accept this approach. Accepting the ambiguity with which decisions must sometimes be made, they understand estimates and that they may be the only way to connect a program with performance measures. They understand the challenge and appreciate the conservative approach, often commenting that the actual value is probably greater than the value presented. When organizations begin to use this routinely, it sometimes becomes the method of choice for isolation.
Data Collection From Other Experts
The previous approaches describe how data are collected from participants in the programs. Both the focus group approach and the questionnaire approach can be helpful in collecting data from others. Sometimes the supervisor of program participants may be capable of providing input on the extent of talent development’s role in performance improvement. In some settings, the participants’ supervisors may be more familiar with the other factors influencing performance. Consequently, they may be better equipped to provide estimates of impact. A word of caution: If the supervisors are physically removed from the actual settings, it may be difficult for them to understand the impact of talent development.
Managers may be asked to provide input, but only if they have some credible insight into the cause-and-effect relationship of these factors. If they are physically removed from the situation, they may not be very credible. Other possible sources of contributions include input from customers, external experts, the program sponsor, and any other group or individual who may be knowledgeable of these relationships.
Building Credibility With the Process
Isolating program effects is the most significant credibility step in the ROI Methodology. It is important to look more closely at issues around selecting techniques and strengthening credibility of the technique.
Selecting the Techniques
Table 4-3 shows the frequency with which each technique was selected as being applied by more than 200 best practice organizations that have been applying the ROI Methodology for five years or more. This table presents a high percentage level for comparison group analysis; the average use of this method in all impact studies would be significantly less. After all, these are best practice organizations, and they have worked diligently to use the most credible analyses. The 20 percent representing “Other” is a variety of techniques that are less likely to be used.
Isolating the Effects of Talent Development Programs
|Method1||Best Practice Use2|
1. Control group arrangement
2. Trend line analysis
3. Expert estimation
1. Listed in order of credibility.
2. Percentages exceed 100 percent.
With several techniques available to isolate the impact of learning, selecting the most appropriate techniques for the specific program can be difficult. Estimates are simple and inexpensive, while others are more time consuming and costly. When attempting to make the selection decision, several factors should be considered:
• feasibility of the technique
• accuracy provided with the technique, when compared to the accuracy needed
• credibility of the technique with the target audience
• specific cost to implement the technique
• the amount of disruption in normal work activities as the technique is implemented
• participant, staff, and management time needed with the particular technique.
Using Multiple Techniques
Multiple techniques or sources of data input should be considered because two sources are usually better than one. When multiple sources are used, a conservative method is recommended to combine the inputs. A conservative approach builds acceptance and credibility. The target audience should always be provided with explanations of the process and the various subjective factors involved. Multiple sources allow an organization to experiment with different techniques and build confidence with a particular technique. For example, if management is concerned about the accuracy of participants’ estimates, a combination of a control group and participants’ estimates could be attempted to check the accuracy of the estimation process.
It is not unusual for the ROI in talent development to be high. Even when a portion of the improvement is allocated to other factors, the numbers are still impressive in many situations. The audience should understand that, although every effort was made to isolate the impact, it is still a figure that is not precise and may contain error. It represents the best estimate of the impact given the constraints, conditions, and resources available.
One way to strengthen the credibility of the ROI is to consider the different factors that influence the credibility of data. Table 4-4 is a listing of typical factors that influence the credibility of data presented to a particular group. The issue of isolating the effects of the talent development program is influenced by several of these credibility factors.
The reputation of the source of the data is important to consider. The most knowledgeable expert must provide input and be involved in the analysis in this topic. Also, the motives of the researchers can have a major influence on perceived credibility. A third party must facilitate any focus group that is done, and the data must be collected objectively. In addition, the assumptions made in the analysis and the methodology of the study should be clearly defined so that the audience will understand the steps taken to increase the credibility. The type of data focuses directly on the impact data: The data have changed, and the challenge is to isolate the effects on that change. Managers prefer to deal with hard data, typically collected from the output of most programs. Finally, by isolating the effects of only one program, the scope of analysis is kept narrow, enhancing the credibility.
• Reputation of the source of the data
• Reputation of the source of the study
• Motives of the researchers
• Personal bias of audience
• Methodology of the study
• Assumptions made in the analysis
• Realism of the outcome data
• Type of data
• Scope of analysis
Getting It Done
In chapter 2, you were introduced to the data collection plan and the ROI analysis plan. In chapter 3, you completed the data collection plan for a program you plan to evaluate to ROI. Here is where you begin completing the ROI analysis plan.
Table 4-5 provides a blank ROI analysis plan. Transfer your Level 4 measures from your data collection plan to the first column of the ROI analysis plan. Then, identify the techniques you will use to isolate the effects of the program from other influences and write the techniques in the second column aligned with each Level 4 measure. Remember, this step must be taken, so a technique should be included for each objective.
In the next chapter, you will continue completing the ROI analysis plan.