Large language models for generating key-feature questions in medical education

Main contribution

This descriptive study evaluated whether OpenAI’s o3 model could generate key-feature questions aligned with Medical Council of Canada guidance. Using a structured prompt, the authors generated 20 cardiology KFQs from recent ESC guidelines; expert review found 93.7% checklist compliance, with 3 accepted as is and 17 accepted with minor revisions.

Practical use

Educators may use a structured LLM prompt to create first drafts of KFQs, especially when building cases across large content areas. The article supports using AI to reduce drafting burden, not to replace clinical and assessment review.

Main caution

The generated KFQs still had clinically important errors: weak or implausible distractors, missing or incorrect “killer” responses, and occasional inaccuracies such as a pregnancy test option for a male patient. Expert review remains necessary before use.

Research gap

The study tested only 20 English-language cardiology KFQs generated with o3. It did not test learner performance, reliability, discrimination, validity impact, other specialties, other languages, or lower-performing models.

Prompt / method

Includes prompt

Yes