AI MedEd Observatory

What Can AI Do in Medical Education?

Evidence, prompts, checklists, and practical interpretation for assessment, feedback, scoring, and teaching materials.

A curated resource for medical educators and researchers who want to understand how AI is being used and what the evidence suggests.

Explore the Observatory Browse Evidence Map

Curated by Yavuz Selim Kıyak, MD, PhD

As a researcher deeply involved in integrating AI into medical education, with several publications in Medical Teacher, Medical Education, and Academic Medicine, I created this personally curated resource because the literature on AI in medical education is expanding rapidly, making it increasingly difficult to identify what is useful, trustworthy, and applicable. This site organizes selected evidence and shares what I am learning. It is to help educators and researchers.

Mantra

Evidence before automation. Judgment before adoption.

Main question areas

Practical starting points

The observatory is organized around common educational questions rather than around AI tools.

4 subquestions5 linked articles5 published prompts

What can AI do for generating assessment materials?

Explore area

1 subquestions2 linked articles0 published prompts

What can AI do for automated scoring?

Explore area

1 subquestions2 linked articles0 published prompts

What can AI do for feedback?

Explore area

Why this site exists

Practical interpretation, not a generic tool directory

Many articles now report AI-generated questions, feedback, scoring, and educational materials. This site organizes selected studies into practical, evidence-informed summaries so educators and researchers can judge what may be useful, what requires expert review, and what remains uncertain.

Curator note

Human review remains essential. Use as a starting point.

Caution

Current evidence suggests some uses may help with drafting, structuring, or calibration. Evidence is limited, and many applications are not ready for unsupervised summative use.

Trust this resource?

Context for curation

This Observatory is curated by Yavuz Selim Kıyak, MD, PhD. His work in this area was noted in an independent bibliometric analysis of ChatGPT in medical education, which identified him as the most prolific author in the field. This is shared only as context for the curation of the resource, not as a claim of authority. The value of this Observatory lies in its transparent organization of the literature, practical interpretation of published evidence, and direct links to the original studies, allowing readers to form their own informed judgments.

Latest additions

Recently added articles and prompts

Recent entries help show where the observatory is currently growing.

Article

Added July 19, 2026

Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items

This innovation report presents an evaluated human-in-the-loop workflow for generating Concordance of Professional Judgment items from a locally defined ethics framework. ChatGPT-5 Thinking and Gemini 2.5 Pro each generated 28 Turkish items; after independent faculty review and adjudication, 25/28 and 27/28 items, respectively, were accepted or required only minor revision.

Read article summary

Article

Added July 5, 2026

Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination

This cross-sectional study compared OSCE scores from three human evaluators and two AI systems, ChatGPT-4o and Gemini Flash 1.5, for 196 pre-clinical medical students performing four clinical skills. AI scores were generally higher than human scores, with better alignment on visually observable steps than on auditory or communication-dependent criteria.

Read article summary

Article

Added July 5, 2026

Validity of AI-generated multiple-choice questions in medical education: a systematic review

This review organizes the evidence on AI-generated MCQs in medical education using Messick’s validity framework. It shows that evidence is strongest for content review, but weaker for response processes, psychometrics, links with other assessments, and consequences.

Read article summary

Creative Commons Attribution 4.0 International License (CC BY 4.0)

Concordance of Professional Judgment item generation prompt template

Added July 19, 2026

Prompt

Your role: You are an assessment designer creating Concordance-of-Professional-Judgment (CoPJ) items for health-profession learners. Items must reflect authentic ethical/professional dilemmas where more than one response can be defensible and expert opinions can diverge. Include expert explanations and a concluding key message.

Inputs (fill these before running):

• TOPIC: << e.g. reporting treatment errors; respect for confidentiality; breaking bad news; legal compliance; bedside >>.

• LEVEL: Undergraduate medical students.

• N_ITEMS: 1

• N_EXPERTS: <<e.g., 6–8; aim for several to show perspective diversity>>.

• LOCALE / POLICY HOOKS (optional): <Türk Tabipler Birliği Etik Bildirgeleri>.

• LANGUAGE: Turkish

Global requirements:

1. Vignette realism & brevity: Less than 50 words; clearly stated setting, role, and constraint; leave some ambiguity. Put the learner into the role. (e.g. “You are …”)

2. Response task: Present one specific behavior/attitude that a learner could take next. The learner will rate the behavior/attitude on a 4-point forced Likert scale:

Totally unacceptable | Unacceptable | Acceptable | Totally acceptable. Do not allow a neutral option.

3. Panel response: Provide an N_EXPERTS distribution across the four options for each behavior (e.g., 5/8 Totally unacceptable, 2/8 Unacceptable, 1/8 Acceptable).

4. Expert explanations: Include one short justification per expert (1-2 sentences each). Ensure credible disagreement across experts and make reasoning transparent.

5. Key message (educational synthesis): Conclude each item with a concise “Key message” as a narrative that includes:

o key principles,

o contextual nuance,

o cites relevant local policy/ethics codes, and

o offers 2–3 actionable “remember” bullets.

6. Tone & ethics: Respect patient dignity, colleagues, and uncertainty. Avoid implying a single “right answer”; emphasize reasoned judgment under uncertainty.

7. Length guardrails: Keep each item (vignette + behaviors + panel + explanations + key message) under ~450 words.

Structure to produce for the item:

1. Situation (vignette).

2. Behavior/attitude option (one), to be rated on the 4-point scale (no neutral).

3. Panel responses (counts out of N_EXPERTS per behavior), in bullet points.

4. Expert explanations (short, diverse justifications).

5. Key message (educator’s narrative synthesis with policy hooks).

Output Format (user-friendly prose layout):

Title: Concordance-of-Professional-Judgment Item

Topic: <Topic provided by the user>

Learning Objective: [A clear, one-sentence learning objective]

Vignette: [A short, context-inclusive, and unambiguous scenario of no more than 50 words]

Behaviour/Attitude: [A specific behaviour/attitude a learner could take next]

Question:

This behavior/attitude is:

☐ Totally unacceptable

☐ Unacceptable

☐ Acceptable

☐ Totally acceptable

Panel Feedback:

[List of 6–8 experts with name only with initials, role, Experience: years in practice, selected option, and justification]

Concordance Summary:

Totally unacceptable: x/x X%

Unacceptable: x/x X%

Acceptable: x/x X%

Totally acceptable: x/x X%

Educational Synthesis (as a narrative):

Key principle(s), contextual nuance, suggest consulting local policy / code of ethics, and finally offer 2–3 actionable “remember” bullets.

Tags: [relevant tags]

Quality checklist:

• Ambiguity present; multiple defensible positions; no single correct answer implied.

• Forced 4-point scale used; no neutral.

• Divergent expert opinions represented credibly.

• Explanations are transparent (cognitive apprenticeship feel).

• Key message cites or invites local policy/ethics when relevant (e.g., disclosure duties).

• Lengths within limits; respectful, professional tone.

Notes: Original full prompt evaluated with ChatGPT-5 Thinking and Gemini 2.5 Pro to generate CoPJ item drafts across 28 topics in the Turkish Medical Association's ethical declarations. Intended for formative content development with local parameter adaptation and expert review.

Sources: Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items, Supplementary Material 1 (full prompt)

Related subquestions: What can AI do for generating Concordance of Professional Judgment items?

Open prompt page

Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License

Medical MCQ generation prompt template (case-based)

Added July 5, 2026

Prompt

You are developing a question bank for medical exams focusing on the topic of [PLEASE INSERT A TOPIC]. Please generate a high-quality single best answer multiple-choice question. Follow the principles of constructing multiple-choice items in medical education. Generate the questions using the following framework:

Case (write as a single narrative paragraph without providing each part separately):

    Patient details (gender/age)
    Presenting complaint
    Relevant clinical history
    Physical examination findings
    Diagnostic test results (optional)

Question stem: [Insert relevant information from the above sections without compromising the answer]

Acceptable question style: Ask for the BEST answer, NOT one that is TRUE/FALSE.

Answer options:
    [Insert plausible answer option]
    [Insert plausible answer option]
    [Insert plausible answer option]
    [Insert plausible answer option]
    [Insert plausible answer option]

Explanation:
• Identify and explain the correct answer.
• Explain why this is the most appropriate answer based on evidence-based guidelines or expert consensus.
• Briefly explain why the other answer options are less correct or incorrect.

Notes: Original prompt for generating case-based single-best-answer MCQs in medical education. Adapted into a Custom GPT while preserving the original instructional framework. Intended as a prompt resource rather than evidence of effectiveness.

Sources: Original source, Custom GPT version

Related subquestions: What can AI do for generating MCQs?

Open prompt page