Technology for Measurement and Analytics
Feature Story – By Steven Just, Ed.D., and Jim Delaney
Learning and measurement are inseparable
If you ask many training leaders if they have a learning strategy, you will likely get a positive response. They’ll describe how they do onboarding, new product training, compliance training, management and leadership development, their processes for continuous learning and upskilling, their learning technology stack, etc.
But if you ask them if they have a measurement strategy, you’ll often get a response along the lines of:
“We test our people when we need to, we do surveys after all instructor-led courses, and our learning management system produces lots of data.” But do you have a strategy? “Well, no, we don’t exactly have a strategy.”
Inseparable Allies
Unfortunately, many learning professionals see measurement as an afterthought, unless it aligns directly to a compliance or mandated training requirement. It’s something they realize they should be considering more formally, but it’s rarely seen or treated as a top priority.
The modern learning organization understands that learning and measurement are inseparable. They have evolved to become a single data-driven function that embraces information and insights to partner successfully with their business stakeholders. Furthermore, the combined technologies often utilized in an organization’s modern learning ecosystem can empower learning teams to surface all kinds of meaningful metrics that reach beyond traditional data.
It’s helpful to think about your company’s measurement strategy and its associated technologies as answering three core questions:
- Why measure?
- What should you measure?
- Which technologies and data sources do you need to do this?
Why Measure?
Fundamentally, you can focus your efforts to formulate your measurement strategy in three core categories:
- Operational Efficiency, to drive practical day-to-day operations and decision-making. These are the bread-and-butter metrics of any learning organization to help manage (and forecast) your budget and resources.
- Training Effectiveness, to validate and improve the effectiveness of your learning experiences. Assessments are traditionally thought of as learning checks and/or certifications, which they are, but there is also a large body of research evidence that frequent formative (low-stakes) assessments improve learning retention. These formative assessments can be in the form of quizzes, pre/post tests, adaptive questioning exercises or games and contests. Low-stakes assessments can also be used to diagnose learning gaps and provide targeted remediation. Most organizations also measure learner satisfaction and course “quality” through learner surveys. (Caveat: Research shows that traditional learner surveys – smile sheets – do not correlate with training effectiveness and a modern, scientific approach should be considered.)
- Organizational Impact, to establish strategic alignment and value with the business. These metrics help you create and deliver the business case for learning and development.
What Should You Measure?
The problem we often encounter is not a lack of data. It’s often just the opposite: Multiple platforms are generating an avalanche of individual dashboards and unanalyzed data. The challenge is to figure out what’s important and how it can be synthesized and combined into actionable metrics.
The key is to focus on the data that answers the important questions for your organization. Some common examples are included in Figure 1.
Which Technologies Do You Need?
The good news is that many of the technology platforms training departments use to implement their learning strategies also collect data as a byproduct of their use. The challenge is to understand how to manage these multiple data sources to produce coherent and actionable metrics.
Here are the components of a typical learning and measurement ecosystem (See Figure 2):
Core Learning Platforms
- Learning Management Systems (LMS): The backbone of most learning infrastructures, LMSs have evolved significantly over the past 25 years, from simple systems that managed instructor-led courses and launched e-learning to today’s talent management systems that do everything related to employee management and learning: recruiting, e-learning, assessments, learner surveys, microlearning, social learning, gamification, performance reviews, competency management and analytics. In the process of deploying all this functionality, these LMSs generate and store huge amounts of learning data. At the same time, there are much simpler LMSs that stick to the original mission of launching and tracking courses.
- Survey Systems: Many training organizations use standard, generic survey software for creating Level 1 evaluations. These systems offer all the capabilities you need to perform these evaluations.
- Assessment Systems: There are several stand-alone assessment systems that do everything from simple quizzing to producing valid, high-stakes certification exams. At the same time, some companies use the assessment capabilities of their LMSs for these tasks. Typically, the stand-alone assessment systems are much more feature-rich, but companies that use their LMSs reason that their assessment capabilities are “good enough.” There are also assessment systems that provide tools for viewing and scoring video role plays, sometimes as stand-alone systems, sometimes as part of larger learning platforms.
Specialized Learning Platforms
- Microlearning/Gamification Systems: We’ve lumped these two system types together because there is often overlap: Many microlearning platforms incorporate elements of gamification and many gamification systems use the techniques of microlearning. Both these system types often incorporate some form of questioning/quizzing, either in the form of games and contests or in the form of adaptive questioning exercises.
- Learning Experience Platforms (LxPs): A relative newcomer to the learning ecosystem, LxPs have some of the same capabilities of LMSs and are typically characterized by mobile-friendly learner interfaces, less of an emphasis on assigned and structured curricula, use of AI for recommending courses, social learning and a learner-centric focus rather than a learning administrator focus. LxPs are also capable of tracking external learning, beyond their own offerings. LxPs often store their usage data in Learning Record Stores (LRSs), a more modern way of tracking learning experiences than is offered by the typical LMS.
Other Data Sources
Other enterprise systems and their data sources, not typically owned by the training department, can be critical for making business case arguments (such as ROI). They include sales/revenue tracking systems, HR/employee engagement platforms, customer relationship management systems and data warehouses.
Pulling It All Together
Most training departments deploy more than one of the above learning systems and related platforms, each with its own data repository. So, you need a way to pull these data sources together, which brings us to the final piece of the measurement ecosystem:
- Analytics
- Reporting
- Dashboards
Most ecosystems will require you to pull relevant reports from a variety of systems and interfaces, but a unified learning dashboard should fully integrate the metrics you need to address the three “Whys” previously identified: Operational Efficiency, Training Effectiveness and Organizational Impact.
Measurement and analytics are both art and science. The science part is obvious, but the art is important too: Your learning systems may be producing far more data than you ultimately need – depending on where you are in the maturity of your learning organization – so the art of analytics is figuring out what’s important and reporting on it in a consistent and meaningful way.
You can think about what’s important by asking yourself this question: “What story do I want to tell my organization about the L&D function?”
As learning science and learning technologies continue to evolve and advance, learning organizations need modern measurement strategies and technologies to monitor operational efficiency, improve training effectiveness and drive the business value of learning.
Steven Just, Ed.D., and Jim Delaney are principal consultants at Modern Measurement Excellence. Email them at info@measurementexcellence.com.
