METRICS & ANALYSIS – By Steven Just, Ed.D.
Sometimes you need to dig deeper
Everyone knows how to compute an exam average. We use them all the time; they’re the most common measure of learning effectiveness (along with passing rates). But, while averages and passing rates are easy to compute, what if interpreting them is not so obvious?
Let’s explore two cases to understand how computing and reporting on average scores and passing rates may obscure serious training issues. For our examples, we will assume an exam with a passing score of 85% delivered to two different classes.
- Class A takes the exam and everyone in the class scores exactly 85%, for an average of 85%. Class A has a passing rate of 100%.
- Class B takes the exam and half the learners score 100%, while the other half score 70%, also for an average of 85%. But Class B has a passing rate of only 50%.
Two classes, the same average score but two different results and two very different implications for the training program. Clearly, something has gone very wrong with Class B. You need to dig deeper into the results, and probably the training, to figure out what happened. Why did half the learners perform so poorly? Only reporting the average score is not sufficient.
Retesting
Of course, the testing of Class B does not end here. Most companies provide multiple tries to pass a high-stakes exam. I generally recommend three tries, but this policy varies from company to company.
Let’s assume three tries.
- Class B Retest 1: On Retest 1, half the learners score 80%, while the other half score 100%. We now have an overall passing rate of 75%.
- Class B Retest 2: On Retest 2, the remaining learners all score 100%. We now have the same 100% passing rate for Class B as Class A.
Reporting
Class A reporting is straightforward: 100% passed with an average of 85%.
But what about Class B? If we just report the Class B passing rate, it is 100%. The same (but not really the same) as Class A.
And what about average score? Misleadingly, Class B has a higher average score than Class A! It doesn’t matter if we report only the passing tries of Class B, or all tries, Class B’s average is above 85%. If we report the average of all scores for Class B, their average is 88.5%. If we report only the passing or final tries of Class B, their average is 100%.
So, what can we do to report our results accurately? The solution is to break out the tests and retests (for Class B) and resist the temptation to group them all together into averages. Something like this:
Conclusion
It’s clear from the two tables that, although everyone passed the exam (eventually) in the two classes, the underlying data tells a very different story.
So, the key reporting takeaway is: Don’t allow averages or overall statistics to obscure important results. You need to disaggregate the data to gain accurate and actionable insights.
Steven Just Ed.D. is CEO and principal consultant at Princeton Metrics. Email Steven at sjust@princetonmetrics.com or connect through LinkedIn at linkedin.com/in/steven-just-081b76.
