To measure AI adoption in Learning and Development (L&D) effectively, it is essential to track more than just the number of prompts used. A team that created 40 courses using AI this quarter may seem like a success, but if authors spent twice as long fixing drafts, the efficiency of AI adoption is questionable.
Tracking work from the first use of an AI tool to the finished learning experience is necessary to understand how effectively AI is adopted. This involves tracking who uses AI, for which tasks, how much work remains afterward, and whether the resulting training helps people learn.
Measuring AI Adoption
A simple way to measure AI adoption is to map AI use to specific L&D tasks. AI adoption can cover a wide range of activities, from generating course outlines to answering employee questions in an LMS. By focusing on one workflow at a time, such as creating a course, and marking where AI is used, it is possible to get a clearer picture of its effectiveness.
For example, when using AI to generate voiceovers for product training, measure the time authors spend preparing scripts, generating audio, correcting mispronunciations, and getting the final version approved. Compare this to the time spent recording and editing narration for similar courses to see if AI speeds up the process.
It is also important to track regular use, not just account access. Define an eligible group and track how many used AI on at least one eligible task, how often, and for what purpose. This can provide a more accurate adoption rate, such as the number of active users divided by eligible users.
A monthly check-in with authors, where they tag uses in a lightweight project log, can provide valuable insights into how AI is being used and where it can be improved. This approach is more reliable than counting prompts, as it takes into account the varying ways people use AI for course creation.
Evaluating AI Effectiveness
When evaluating the effectiveness of AI, it is essential to include editing and review time in the calculation. An AI-generated module or quiz might save an hour of writing but add two extra rounds of SME corrections. By tracking the time spent reviewing and revising AI content, it is possible to get a more accurate picture of its impact on the workflow.
For each pilot project, record the time from assignment to an approved, publishable asset, and break it down into specific steps. Compare similar projects, and describe the limits of the comparison. This approach can help identify where AI is saving time and where it may be causing slowdowns.
Asking employees about their experiences with AI can also provide valuable insights. Once a month, ask the team a few short questions, such as which AI task saved them the most work, what took longer than expected, and which outputs they decided not to use. Look for recurring patterns and group the answers by task to identify areas for improvement.
Reporting AI Adoption
A monthly report on AI adoption can fit on one page. Show the eligible groups and projects, the total time to approved output, and one learner or workplace indicator, where available. Include a short note on what the team will change next month.
For example, the report might state that 8 out of 12 authors used AI on 14 of 20 eligible projects. Narration production took less time, but terminology corrections added an extra review round in four modules. Next month, the team will test an approved pronunciation list and check whether review time falls.
The reporting period and definitions should be kept consistent. As the team expands to translation or learner support, each workflow should have its own measures. The results can be brought together later once it is known what successful use looks like in each one.
Leave a Reply