TECHNOLOGY – By Julia Hatherell and Natasha Ilincic
Meet your promising new measurement and evaluation assistant
With global investments in training and development forecasted to grow from$163 billion to $487.3 billion USD by 2030, measurement and evaluation (M&E) is crucial to ensuring the learning industry is doing things right (Business Wire, 2023).
However, demonstrating the return on investment (ROI) of a learning program or collecting data to inform a learning strategy can be challenging. That’s often why M&E is deprioritized – a lack of time, budget and resources to put in the work.
So, is there a way we can make M&E easier to prioritize? Enter artificial intelligence (AI): a promising new measurement and evaluation assistant.
Making Waves
Generative AI is AI that generates text, images or other data in response to prompts, and it can play many roles. It has made waves in the learning and development (L&D) industry by expediting content development … but content and timesaving are only one part of the picture.
In reality, generative AI can be used in many applications, such as building an M&E strategy by:
- Brainstorming a starting point for an M&E strategy.
- Reviewing ideas.
- Strategizing and iterating the approach.
- Designing survey, interview and assessment questions.
- Analyzing measurement and evaluation results.
To illustrate this point, let’s look at a simplified, fictional case study of Micah, an L&D manager who is designing an M&E strategy to support commercial product launch training.
Following Micah, you’ll see how AI can lighten the load that comes with creating and executing your M&E strategies. You’ll discover that AI is not just a tool for increasing efficiency, but also for enhancing decision-making, amplifying creativity and more.
The Power of Prompts
As you may have already experienced with AI, you’ll rarely get exactly what you need in only one prompt. You need to be detailed, strategic and persistent with your prompts to get a valuable result.
It is also crucial to understand how to best protect yourself and your data. You should always refer to your company’s AI and data privacy policy before using a new tool.
- Data Privacy: To be safe, only use a company-sanctioned AI platform, such as an enterprise version of ChatGPT. If your organization doesn’t have one, but you do have a clear AI policy in place, follow that.
- Lack of Context: AI only knows the context that you provide. Be diligent when relying on AI to interpret data.
- Bias and Sources: Keep in mind that AI algorithms rely on historical data to identify patterns and trends, which may introduce bias. Most times, generative AI models rely on information publicly available on the internet. So, you must always think critically about your outputs and whether they are affected by bias.
- Human Reliance: The power of AI begins with a strong input. Human oversight is essential for providing the AI tool with a useful input, and in drawing meaningful conclusions and making informed decisions based on the generated output.
When we think about M&E and AI, we often focus on the results, yet they are only one part of the bigger picture. When you begin to consider how AI can support you in all stages — conception, strategy, design and execution – and learn how to use and prompt AI effectively, the possibilities grow exponentially.
So, let’s continue with Micah’s story!
Setting Objectives
Micah’s first task is to define the learning and business objectives for the new training program.
She turns to AI to brainstorm common learning objectives for a sales training program. Her first few outputs are vague and non-specific.
Micah tries again, this time asking AI to put itself in her shoes and consider what the focus should be for sales representatives in an oncology therapeutic area. The new outputs are better, and after some editing, she ends up with focused objectives that she shares with senior stakeholders overseeing the program.
Next, she uses AI to review and suggest improvements to the goals to ensure they are measurable. She refines and iterates her prompts to develop a set of robust learning and business objectives with defined endpoints and metrics to track.
From there, Micah begins to map out the learning program. She asks AI to recommend an M&E strategy based on her training program goals and uses follow-up prompts to refine the output based on data collection methods she has access to, her discussions with the sales director and what has been done previously.
The recommended strategy involves a mix of key performance indicators (KPIs),assessments, stakeholder interviews and informal feedback gathering. Feeling overwhelmed, Micah turns to AI for help to get started with designing the M&E assessments.
- Stakeholder Interviews: Micah inputs key information, including her learning and business objectives, to return a list of questions she can ask learners in a 30-minute interview. She tweaks the questions to ensure they are personal to the product, therapeutic area and company. During the interview, she uses an AI note-taker to transcribe the discussion. Afterward, she feeds the transcript into her AI tool to pull the key takeaways from each interview.
- Survey Design: Micah uses AI to generate a list of survey questions to be used during and after the training program to evaluate how confidence develops along the way and which elements make the biggest impact. She plugs the results into AI to generate takeaways and a thematized breakdown of the responses. She takes time validating the takeaways to ensure their accuracy, but overall, this method saves her a lot of time.
- Assessment Questions: Micah plugs in some training content into AI and requests a list of assessment questions ranging from multiple-choice product knowledge to selling scenarios. She collects feedback from internal subject matter experts to make them more relevant and specific for the sales force. Then, she asks AI to incorporate their feedback and rewrite the scenarios, producing several strong questions.
- Roleplaying Tool: Micah trains an AI chatbot using only the publicly available product label information and has the sales reps use it to simulate conversations with customers. Not only is this a good practice opportunity for the learners, but it collects real-time insights into where they are getting stuck.
The Results Are In
After a year, the training program is well underway. Micah has launched a post-training assessment survey, has collected KPI data and is now ready to analyze the results.
She inputs the data into AI to conduct an analysis of numerical data and determine statistical significance, evaluate whether her goals were met and generate recommendations for future enhancements and initiatives. The program returns with quantitative results for numerical data and a summary of qualitative results by pulling key takeaways, feedback and quotes from the learners, saving Micah countless hours analyzing the results herself.
Invest the Time
There are many ways that AI can support the full measurement and evaluation process to ensure a focused approach, generate the right data and analytics and demonstrate results. Through critical thinking and iterative design, you can use AI to give you a boost in productivity.
However, AI isn’t perfect, and it isn’t always easy to use. You should expect to invest some extra time up front to learn what types of prompts work best for your needs.
Most importantly, you should always remember the limitations of AI and refresh yourself with your company’s AI and data privacy policies before exploring.
Here are two top tips:
- Set AI up for success. Some AI platforms have the entire internet at their disposal. While this makes for a rich source of information, it doesn’t necessarily translate to valuable output. The more specific and detailed the prompt, the better the output.
For example, tell AI that it is an expert in the area that you need support in, and then be very specific about what you want, providing an example if possible. - Iterate. If AI isn’t doing what you want it to do, don’t be discouraged. You’ll rarely get exactly what you need in only one prompt (if you do, then congratulations – that’s impressive!).
When you receive a response you’re not happy with, identify what’s wrong(too vague, too long, wrong tone, etc.) and then directly ask AI to be better in that specific way.
Conclusion
Remember, learning how to best leverage AI is a journey, but if learning professionals can invest the time up front, they have a real opportunity to advance industry M&E capabilities.
With some practice, iteration and curiosity, AI can introduce efficiencies in your day-to-day, drive decision-making, and expand your creativity and the scope of your capabilities.
Let 2024 be the year you explore AI! We know it’ll be worth the investment.
Julia Hatherell is manager, healthcare marketing & learning strategy, for Metrix Group. Email her at jhatherell@metrixgroup.com or connect through www.linkedin.com/in/julia-hatherell/.
Natasha Ilincic is manager, medical writing & learning design, for Metrix Group. Email her at natasha.ilincic@metrixgroup.com or connect through www.linkedin.com/in/natasha-ilincic/.
