METRICS & ANALYSIS – By Steven Just, Ed.D
There is a way to communicate training’s value
Despite years of effort, the challenge of linking learning and development(L&D) to tangible organizational success remains a tough nut to crack. ATD surveys over a 10-yearperiod show no movement in the number of L&D departments that evaluate training at Kirkpatrick or Phillips Levels Three or higher.
Level Three (Training Transfer) is measured by just over 50% of organizations. Level Four (Organizational Impact) just crosses the one-third threshold and Level Five (ROI) is only attempted by 15% to 20% of L&D departments.
So, the question is why? If we all agree that it’s important, if we feel we need to make the case for L&D to the C-suite, why don’t more learning leaders do it? When surveyed, most learning leaders cite lack of time and budget.
Here’s a different theory: Time and budget may be factors, but the most important factor is that it’s not entirely clear how to do it. We know how to create Level One learner surveys, we know how to create Level Two assessments of learning (even if much of what we do at these two levels does not clear the validity bar), but there is no generally agreed upon methodology for measuring at Level Four.
When conducting a Level Four evaluation, one methodology is the Brinkerhoff Success Case Method (SCM).
Some Background
Robert Brinkerhoff, Ed.D., was a professor (now retired) at Western Michigan University, specializing in program evaluation. More than 25 years ago, he was asked by a well-known supplier of leadership development programs to evaluate the impact of their courses across several corporations.
What Brinkerhoff found was initially puzzling. He discovered that the same curriculum had different outcomes in different organizations. He even found that different business units within the same corporation had different results.
Anyone who has studied the scientific method knows that if you hold a single variable constant (in this case the training) but get different outcomes, that variable cannot be the cause of the different outcomes. It must be another factor (or factors).
Digging further, Brinkerhoff looked at how the trainees applied what they had learned. Here’s what he found:
- 15% did not try to apply the training.
- 70% tried but gave up.
- 15% tried and had some success.
Over hundreds of studies, he discovered that only 15% to 20% of training efforts had any measurable organizational impact. According to Roger Chevalier’s
A Manager’s Guide to Improving Workplace Performance, other studies have shown that about 75% to 85% of organizational performance issues are due to environmental factors (beyond the individual employee and their training).
If you think about it, these are astounding numbers. $100B per year is a generally accepted number for how much U.S. corporations spend on training. If only 20% of that expenditure results in a positive impact, what does that say about the other $80B? How much money is being wasted, not necessarily by “bad” training, but by failing to provide an environment in which the training can be advantageously applied?
The Success Case Method
Based on his and others’ research, Brinkerhoff surmised that successful training was a process, not an event and that the issue was performance, not training perse. He then created the Success Case Methodology to answer two simple questions: When training works, why does it work? And when it fails, why does it fail?
He knew from his research that a single training event with multiple learners, on average, will produce individual successes (15%-20%), individual failures (15%-20%) and everyone else somewhere in the middle. So, his methodology rests on identifying those handful of successes and failures and isolating the factors that lead to success or failure.
Success and failure can be measured by either hard data, such as market share (for sales training) or workplace accidents (for safety training), or soft data, like employee satisfaction (for leadership development) or customer satisfaction (for sales training).
A slightly modified version of the SCM consists of nine steps:
- Step 1: Clarify the goals of the training program (key performance indicators or KPIs). What does success look like?
- Step 2: Conduct the training program.
- Step 3: After an appropriate time, measure results.
- Step 4: Identify high- and low-success cases based upon the observable behaviors and KPIs of Step 1.
- Step 5: Conduct structured interviews with representative members of each group. (I use a performance improvement model to do this.)
- Step 6: Synthesize the data from the interviews. Isolate causes for success and failure. What are the common characteristics of each case type?
- Step 7: Compute impact metrics (where appropriate).
- Step 8: Create the success case report highlighting reasons for high success and low success, and lessons learned. Suggest improvements to the training and performance environment.
- Step 9: Present the final report to key stakeholders.
Benefits of the SCM
The SCM has many benefits:
- Clarifying program goals requires coordination with stakeholders outside L&D.
- It does not require a huge team.
- It does not require a long time to complete (typically 10 to 12 weeks).
- It provides feedback for improving training programs and/or the performance environment.
- Success stories are compelling and easily understood.
- It provides a clear line of sight from training to impact.
Benefit 4 is particularly important. Unlike other evaluation methodologies, it doesn’t just produce a number, it produces actionable data that can be used to improve the training and/or the performance environment.
Conclusion
Though lack of time and budget might be the most often cited excuses for not conducting impact studies, consider the possibility that it’s also a lack of a clear methodology. There is a practical solution.
Steven Just Ed.D., is CEO and principal consultant at Princeton Metrics. Email Steven at sjust@princetonmetrics.com or connect through LinkedIn at linkedin.com/in/steven-just-081b76.
