Evidence summary

A systematic review of 71 medical education studies found that AI-generated MCQs are feasible, fast, and often acceptable after review, but validity evidence remains incomplete. AI-generated items often show moderate difficulty, but discrimination is modest. Non-functional distractors are common, meaning some answer options are rarely chosen and do not test effectively. Reported factual or content error rates varied widely, from less than 1% to 45%. Nearly half of studies relied only on expert review, without testing items on learners. AI can produce major time and cost savings, but the savings mainly occur in drafting.

Practical starting points

• Start with a blueprint, learning objective, source content, target learner level, and item-writing rules.
• Ask AI to generate structured draft items, including correct answer and rationale.
• Have subject experts check factual accuracy, relevance, cognitive level, and answer defensibility.
• Review stems and distractors for ambiguity, cueing, implausibility, cultural mismatch, and option-pattern bias.
• Pilot items with learners where possible.
• Analyze difficulty, discrimination, distractor function, and reliability.
• Revise, approve, retain, or remove items based on evidence.