In order to continually improve our learning & development designs and deployments, we frequently look to successful participants, members of our team or the industry to try to understand what learning strategies we can glean from their success.
This is a strategy I am often asked to execute when creating coaching or sales training programs for clients. Certainly there is a good deal to be learned from the most successful in any organization, however isolating them as the only source of “best practices” is a very limited and potentially dangerous perspective. There are critical lessons to be learned from those who fail and those who may have moved on. It is sometimes difficult to recognize situations where survivorship bias affects our decision making negatively.
I find the stories about very smart people using math, technology and brainpower to defeat the enemy during WWII fascinating. One of the lesser known stories that has offered valuable business insights for me is that of Abraham Wald.
Wald was a brilliant Hungarian mathematician, grandson of a rabbi and son of a kosher baker, who was forced to immigrate to the United States in the late 1930s as Nazi Germany’s influence made life very difficult. Wald was invited to Columbia University and fought his war as part of the Statistical Research Group (SRG) think tank. SRG was the mathematical equivalent of the Manhattan project, but using statistics as opposed to explosions. The group of mathematicians solved some of the most important problems facing the Allied war effort: Where guns should be placed on planes, fuel mixtures for optimal distances, torpedo barrages, bombing patterns and numerous other problems that require mathematical expertise to be applied to real-life situations.
One of the most interesting dilemmas was where to place armor in order to protect bombers and at the same time reduce the weight of the plane in order to gain distance. The optimal amount of armor was relatively simple to work out but the placement of that armor was the dilemma. The Air Force studied all the returning bombers to determine where to place the armor. The results were less than effective. The military turned over the data and the responsibility to the SRG and Wald.
It seems that the damage wasn’t evenly distributed throughout the planes; there were more bullet holes in the fuselage compared to the engine area. The military had incorrectly assumed that the armor (and its weight) was more effectively used on the fuselage. Wald’s approach was much different: Put the armour where the bullets aren’t. When he considered the bombers that did not make it back from bombing runs, Wald decided the solution was to armour the engines. The military never considered the bullet holes in the engine blocks they did not observe.
We may not have Abraham Wald and the big brains of SRG to help solve our learning and development challenges but we can certainly learn from survivorship bias. Modeling learning programs after your best employees only is a perfect example of how survivorship bias can be an issue.
If the planes that didn’t return are key to solving the problem for Wald, who are those planes that should be considered in a learning and development strategy?
When we give feedback during a roleplay or coaching session we often start with what the participant did right. It’s not because we want to sugar coat constructive feedback with something nice first. The feedback is important because very often the participant does not always know they’ve done something right. Success can also be a result of luck or good timing. A stupid decision in the moment that turns out well is often a brilliant strategy in hindsight. The more valuable insight from success is what not to do and that requires a failure to work from.
When we analyze why participants do not utilize the training they’ve received and you’ve invested in (the planes that did not return), we find that there are three main reasons:
- Category One: Participants are successful using their own methods and do not use the training.
- Category Two: Participants try to use the training, experience failure, give up and go back to previous methods.
- Category Three: Participants use the training persistently, change and are successful.
Research and industry experts tell us that 10 percent to 15 percent of learning content delivered in formal learning (classroom, eLearning, etc) is retained successfully. This becomes a very low return on investment and makes category three the least effective of the three. The percentage between category one and two may not be relevant and varies from program to program, in my experience. Consider the participants in these categories (the planes that did not return) as the key strategic ingredient in how to create and deliver learning & development successfully.
Before we explore how to extract this critical data let’s look at why 85 percent of participants fall into categories one and two.
The Knowledge-Skill Gap
There are effectively two parts to learning a new skill. The first step is acquiring knowledge. Reading a book, attending a workshop or participating in an eLearning program are common knowledge transfer vehicles for learning & development organizations. The second step is converting that knowledge into a skill. Most learning organizations subscribe to the Center for Creative Leadership’s (CCL) 70:20:10 ratio: 70 percent of learning takes place on the job through practice or experience; 20 percent takes place in informal learning situations like observing others, and 10 percent takes place in formal learning (workshops, reading, eLearning), the knowledge transfer step. Interestingly enough according to the Association for Training Development’s 2014 State of the Industry report, 55 percent of L&D budgets are targeted at formal learning programs, the 10 percent.
Corporate learning & development departments are very good at delivering knowledge (the 10 percent) to a large number of participants in a variety of ways. The low retention and success rates in category one and two are a direct result of a lack of investment in the 70 percent. The 70 percent is left to unfunded managers to “make the training stick.” Most managers are far too busy to place practice and coaching as their highest priority. Most managers are not trained L&D facilitators, and do not understand all of the content. In fact, many sat through the same workshop everyone else did and, like everyone else, only retained at best 15 percent. Also consider that it is very uncomfortable and unprofitable to practice new skills on clients.
Creating Safer Skies
Participants from category one and two are the evidence (the missing planes) we are not considering during the learning and development strategy and design process. Where to place the armor, i.e. where to apply your budget for the greatest return, is CCL’s 70 percent.
The time to address retention is during design, the place to address it is in the field, the method to address it is practice. Creating safe and deliberate practice in the field is the solution to preventing Survivorship Bias from working against you objective.
- Create practice based scenarios that break down the process into small steps.
- Repeat the process and increase skill and intensity.
- Use professional roleplayers to simulate reality and increase emotional intensity.
- Create trust by providing feedback and coaching throughout the practice.
- Measure a defined set of skills.
Bring More Learners Back from Bombing Runs
Step 5 of implementing a practice holds the key to extracting the data to where to place the armor. The professional roleplayer measures each skill during the roleplay so that you end up with relevant cohort data on each skill. Data from the lowest two or three performing skills are then fed back into the knowledge transfer stage (workshop, eLearning) to be reinforced. This feedback loop helps to avoid survivorship bias, and tells you where to put your armor.
The advantage to 1:1 scenario-based practice is that you have the opportunity to talk to every participant.
Abraham Wald and the big brains of SRG helped to win WWII in their own way; they fought with math and logic. Although it is not “life or death,” the battle for budget and resources in corporate learning and development is ours to win. We have a responsibility to our participants to arm them with the best training and practice so they can achieve their objectives in the field.
Randy Sabourin is the co-president of e-roleplay Inc. and co-founder of Anderson Sabourin Consulting Inc (ASCI). Email Randy at rsabourin@e-roleplay.com.
