An Introduction to Training Measurement
Feature Story – By David Vance
To track success, training should measure effectiveness, efficiency and outcomes.
Talent Development Reporting principles (TDRp) is an industry-led effort to bring best practices and principles to the field of learning & development (L&D). The starting point is a framework for measurement and reporting that can provide a common language to use, just as accountants and other professionals have in their fields.
The TDRp framework consists of three types of standard measures and three standard reports. The three types or categories of standard measures are effectiveness, efficiency and outcomes. The best practice recommendation is that all three types of measures should be employed. The three types of reports are operations, program, and summary, and here again, best practice is to use all three types.
The goal was to keep the framework simple and easy to use. We also wanted to build on the excellent work done in our profession over the last 70 years, especially that done by the Association for Talent Development (ATD, then called ASTD) and by Don (now Jim and Wendy) Kirkpatrick and Jack and Patti Phillips. In this article, we will focus on the three types of measures.
Efficiency Measures
Efficiency measures are about the number of courses, participants and classes, as well about utilization rates, completion rates, costs and reach – to name the most common. Typically, these measures by themselves do not tell us whether our programs are efficient or not. Rather we need to compare them to something else, which may be last year’s numbers, the historical trend, benchmark data or the plan (target) we put in place at the beginning of the program.
Now we have a basis to decide if we are efficient and if there is room to improve, to become more efficient. All organizations have efficiency measures with the most common being number of participants, classes and hours, as well as some cost measures such as cost per learner or cost per hour. These are foundational and the basic efficiency measures should be part of any measurement strategy.
Effectiveness Measures
Effectiveness measures are those that address the quality of the learning program or initiative. Effectiveness or quality measures provide a natural balance to efficiency or quantity measures. In learning, we are fortunate to have the four levels popularized by Don Kirkpatrick and the redefined fourth level and the fifth level, return on investment (ROI), popularized by Jack Phillips. These five levels all speak to the quality of the learning.
Level 1 measures the participant’s or sponsor’s satisfaction with or reaction to the learning – certainly an initial measure of quality. Level 2 measures the amount learned or transference of skills or knowledge, and level 3 measures the application of that knowledge or change in behavior. If they didn’t learn anything or if they don’t apply what they did learn, we may not have a quality program. (This may reflect a lack of engagement or reinforcement by the sponsor, but we still have a problem.)
Level 4 measures impact or results and, since this was the reason for undertaking the learning to begin with, if there are no results it is hard to argue we had a quality program. Last, ROI provides a return on our investment, a final confirmation of quality assuming we have properly aligned the learning to our organization’s needs and achieved high quality on the previous four levels.
Most organizations have level 1 and 2 measures, but relatively few measure at the higher levels. Best practice is to measure at least level 3 for all important programs and levels 4 and 5 for key programs, especially those in direct support of organizational goals. So, our greatest opportunity for improvement here is to measure more level 3 application.
Outcome Measures
That leaves outcome measures. Unlike efficiency and effectiveness measures, most organizations are not measuring outcomes, and few even talk about them. That is unfortunate because outcome measures are the most important of the three types, especially to senior leaders who make the funding decisions for L&D. In accounting, this would be similar to reporting expenses and liabilities, but never talking about revenues or assets. No one would have a complete picture of what we do, and it would be hard for anyone to understand why we have the expenses and why we incur the liabilities.
So, what are these all-important outcome measures? Simply put, outcomes represent the impact or results that learning has on your organization’s goals or needs. Ideally, this is the isolated impact of Phillips level 4, so level 4 will do double duty as both an effectiveness measure and an outcome measure. Suppose a needs analysis indicated that your salesforce would benefit from a consultative selling skills program and product features training. And suppose that you and the head of sales agree that it makes sense, and the two of you further agree on program specifics, including mutual roles and responsibilities, especially how the sponsor will reinforce the learning and hold her own employees accountable.
Now, what impact on sales can we expect from this training? How much of a difference can training make? A lot? A little? Enough to be worthwhile? This is the realm of outcome measures, which will sometimes be subjective but very important nonetheless.
Best practice is to begin to focus our attention on outcome measures. These are truly what our CEOs and CFOs would like to see from us but are not getting today, which is why we have highlighted them as a separate category. Outcome measures speak directly to the value of L&D. You should have an outcome measure whenever you can directly support an organization goal like higher sales.
Calculating the impact will require you to isolate learning from other factors, but the Phillips model has provided numerous industry standard methods to accomplish this. If isolation is a problem, the more readily available level 3 application rate may be used as a proxy for the outcome measure. In any case, with outcome measures we are at last focused on how we align with corporate goals or needs and what type of impact we can have, and this is what your senior leaders have been waiting for.
Conclusion
In conclusion, best practice today calls for adding levels 3-5 to your existing measurement strategy with a special focus on level 4 isolated impact, which doubles as the all-important outcome measure. As a profession, let’s start talking about these three types of measures. It would be a big step forward if we could just adopt a common language – which, by the way, is a precondition to be a true profession.
David Vance is executive director of the Center for Talent Reporting and former chief learning officer of Caterpillar. Email Dave at dvance@centerfortalentreporting.org.
