How to Review AI-Generated Key-Feature Questions

How to review AI-generated key-feature questions?

Use "Kiyak’s quality checklist for evaluating Key-Feature Questions", which has been published in a peer-reviewed article, to help educators decide whether AI-generated key-feature questions have a sufficient quality

Purpose: Use "Kiyak’s quality checklist for evaluating Key-Feature Questions", which has been published in a peer-reviewed article, to help educators decide whether AI-generated key-feature questions have a sufficient quality

Source: Original source

Copyright / reuse status: Creative Commons Attribution-NonCommercial License 4.0

Copyable resource

Kiyak’s quality checklist for evaluating Key-Feature Questions

Resource

Kiyak’s quality checklist for evaluating Key-Feature Questions

Please review the KFQ item components based on the criteria below derived from MCC Guidelines (Aug 2012, v3: https://mcc.ca/wp-content/uploads/CDM-Guidelines.pdf). For each criterion, indicate Yes (Y) or No (N). [WI: Write-In, SM: Short-Menu]

Titles	No	Criterion	Yes (Y) or No (N)
A. Key Features (KFs) Definition	1	KFs represent critical, essential, or challenging aspects of the problem, not trivial aspects?
A. Key Features (KFs) Definition	2	KFs focus on steps where learners are likely to make errors or face difficulty?
A. Key Features (KFs) Definition	3	KFs are mutually independent and non-overlapping?
A. Key Features (KFs) Definition	4	KFs state the given clinical context?
A. Key Features (KFs) Definition	5	Each KF contains only one concrete clinical action?
A. Key Features (KFs) Definition	6	The difficulty of the KFs matches the target candidate’s training level?
B. Case Scenario	7	Scenario is realistic and clinically plausible?
B. Case Scenario	8	Scenario puts the candidate into their clinical role? For example, “You are asked to see…” rather than “A patient presents…”
B. Case Scenario	9	Scenario provides sufficient information for the first question without excessively cueing answers to later questions?
B. Case Scenario	10	Scenario avoids jargon and uses lay terms appropriately?
C. Test Questions & Format	11	Scenario is necessary to answer the questions?
C. Test Questions & Format	12	Each question assesses one or more of the defined KFs?
C. Test Questions & Format	13	Tasks and questions are phrased in active, decision-making language? For example, “What would you do …?” instead of “What should be done …?”
C. Test Questions & Format	14	Questions focus on the decision/action itself, not the reasoning behind it?
C. Test Questions & Format	15	Chosen response format, WI or SM, is appropriate for the skill/knowledge being assessed by the question?
C. Test Questions & Format	16	Response limits are clearly stated? For example, “List only one” or “Select up to X.”
D. Short-Menu (SM) Options, If Applicable	17	Option list is alphabetical and contains an appropriate number of choices, for example 10–20?
D. Short-Menu (SM) Options, If Applicable	18	Option list includes “None”/“Do nothing” as the last option?
D. Short-Menu (SM) Options, If Applicable	19	Each option includes a single action only, with no double-barrelled options such as “Give antibiotic and discharge”?
D. Short-Menu (SM) Options, If Applicable	20	Distractors are plausible, relevant, and non-trivial?
E. Scoring Key	21	Scoring key directly and accurately reflects only the defined KF(s) assessed by the question?
E. Scoring Key	22	Each KF contributes 1.0 point total?
E. Scoring Key	23	Equal weighting is used, avoiding differential weighting?
E. Scoring Key	24	Correct answers are accurately listed?
E. Scoring Key	25	If any, “killer” answers/options resulting in a score of 0 are genuinely dangerous or harmful in context?
E. Scoring Key	26	For write-in questions only, appropriate synonyms/equivalences are included?
F. Overall	27	Assessment primarily evaluates clinical decision-making/reasoning, not just knowledge recall?
F. Overall	28	Recommendation: [ ] Accept As Is, [ ] Accept with Minor Revisions, [ ] Requires Major Revisions, [ ] Reject

Related evidence

Linked articles

2026Original research

Large language models for generating key-feature questions in medical education

Yavuz Selim Kıyak, Stanisław Górski, Tomasz Tokarek, Michał Pers, Andrzej A Kononowicz · Medical Education Online

This descriptive study evaluated whether OpenAI’s o3 model could generate key-feature questions aligned with Medical Council of Canada guidance. Using a structured prompt, the authors generated 20 cardiology KFQs from recent ESC guidelines; expert review found 93.7% checklist compliance, with 3 accepted as is and 17 accepted with minor revisions.

Related published prompts

Linked prompt architectures

Creative Commons Attribution-NonCommercial License 4.0

Prompt for generating key-feature questions (problems) in medical education

Prompt

Role & goal
You are a medical-education assessment designer. Create a fully-specified Key-Feature (KF) problem for a Clinical Decision-Making / Reasoning (CDM) examination that follows the Medical Council of Canada’s Guidelines for the Development of Key-Feature Problems & Test Cases (2012).
Input you will receive
•	Clinical problem or objective(s)
•	Candidate level (e.g., graduating medical student)
•	Patient age group (choose one: Pregnancy, neonatal and infant, pediatrics (1-11 years), Adolescent (12-18 years), Adult (19-64 years), and Geriatrics (65 and older)
•	Clinical situation (choose one: undifferentiated complaint / single typical problem / multiple or multisystem problem / life-threatening event / preventive care & health promotion)
•	Site of care (outpatient, inpatient, emergency department, etc.)
Tasks (follow in order)
1.	Problem header – candidate (test taker) level, state problem/objective(s), life-span period, clinical situation, site of care.
2.	Key-features list – write 2 or 3 concise KF statements, each containing:
o	initial condition beginning “Given …”,
o	a single critical task/action or avoidance step,
o	optional qualifier (e.g., “as a leading diagnosis”, “immediately”).
Use specific verbs (order, include, prescribe). No compound tasks.
3.	Case scenario – 80–120 words (longer only when interpretation or management is being tested). Use real-life, lay language; include only details needed before the first question. Write from the candidate’s perspective (e.g., “You are asked to see…”) to embed them in their clinical role.
4.	Exam questions – one per KF (max 3 total).
o	Choose Write-In (WI) format for diagnosis/management questions; Short-Menu (SM) for history, physical findings or investigations.
o	Append the appropriate response-limit clause: “List only one”, “List up to x”, or “Select as many as are appropriate”.
o	For SM questions supply a 15-20-item A-to-Z option list, one action per option, ending with “None”. Include plausible distractors, at least one neutral correct option, and keep medical terms alphabetised.
o	Phrase each question in active, decision‑making language (e.g., “What would you do next…?”).
o	Focus on the clinical decision/action itself, not on the reasoning behind it.
5.	Scoring key – allocate 1.0 mark per KF (equal weighting of multiple correct responses). Identify synonyms for WI answers and specify any “killer” responses that earn 0 regardless of other selections.
Output format
Clinical problem or objective(s)
Candidate level: 
Patient age group:
Clinical situation:
Site of care:

Key Features
KF-1 …
KF-2 …

Case Scenario
…

Questions
Q1 (WI)…        [Limit: List only one]
Q2 (SM)…        [Limit: Select up to X]
…

Scoring Keys
KF-1: …
KF-2: …

Style rules
•	Write questions so they cannot be answered without the scenario.
•	Use concise, directive wording; avoid medicalese in the scenario.
•	Ensure the item primarily assesses clinical decision‑making / reasoning rather than simple knowledge recall.
•	Do not reproduce this prompt
Clinical problem: Primary hypertension
Candidate level: Graduating medical student
Patient age group: Adult (19-64 years)
Clinical situation: Single typical problem
Site of care: Outpatient
**MUST**: Take 2024 ESC Guideline as a reference.

Source: Kıyak, Y. S., Górski, S., Tokarek, T., Pers, M., & Kononowicz, A. A. (2025). Large language models for generating key-feature questions in medical education. Medical Education Online, 30(1), 2574647.

Sources: Large language models for generating key-feature questions in medical education

Related subquestions: What can AI do for generating key-feature questions?

Open prompt page