AI MedEd Observatory

What Can AI Do in Medical Education?

Evidence, prompts, checklists, and practical interpretation for assessment, feedback, scoring, and teaching materials.

A curated resource for medical educators and researchers who want to understand how AI is being used and what the evidence suggests.

Explore the Observatory Browse Evidence Map

Curated by Yavuz Selim Kıyak, MD, PhD

As a researcher deeply involved in integrating AI into medical education, with several publications in Medical Teacher, Medical Education, and Academic Medicine, I created this personally curated resource because the literature on AI in medical education is expanding rapidly, making it increasingly difficult to identify what is useful, trustworthy, and applicable. This site organizes selected evidence and shares what I am learning. It is to help educators and researchers.

Main question areas

Practical starting points

The observatory is organized around common educational questions rather than around AI tools.

4 subquestions5 linked articles5 published prompts

What can AI do for generating assessment materials?

Explore area

1 subquestions2 linked articles0 published prompts

What can AI do for automated scoring?

Explore area

1 subquestions2 linked articles0 published prompts

What can AI do for feedback?

Explore area

1 subquestions1 linked articles1 published prompts

What can AI do for generating scenarios?

Explore area

Why this site exists

Practical interpretation, not a generic tool directory

Many articles now report AI-generated questions, feedback, scoring, and educational materials. This site organizes selected studies into practical, evidence-informed summaries so educators and researchers can judge what may be useful, what requires expert review, and what remains uncertain.

Curator note

Human review remains essential. Use as a starting point.

Caution

Current evidence suggests some uses may help with drafting, structuring, or calibration. Evidence is limited, and many applications are not ready for unsupervised summative use.

Trust this resource?

Context for curation

This Observatory is curated by Yavuz Selim Kıyak, MD, PhD. His work in this area has been noted in two independent bibliometric analyses. A bibliometric analysis of ChatGPT in medical education identified him as the most prolific author in the field, and a bibliometric mapping study of AI for assessment and feedback in medical education, covering 435 records from 2015 to 2026, found that he showed the strongest weighted publication profile among the authors analyzed. This is shared only as context for the curation of the resource, not as a claim of authority; bibliometric visibility reflects publication volume and timing, not the correctness of any finding. The value of this Observatory lies in its transparent organization of the literature, practical interpretation of published evidence, and direct links to the original studies, allowing readers to form their own informed judgments.

Latest additions

Recently added articles and prompts

Recent entries help show where the observatory is currently growing.

Article

Added September 20, 2026

Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education

This matched, source-blinded expert evaluation found that PBL scenarios generated from the same learning objectives with a structured one-shot prompt received a higher overall proportion of positive checklist ratings than existing faculty-authored scenarios: 90.7% versus 80.3%. The strongest descriptive differences favored the AI scenarios for cognitive load balance, consistency, scientific or clinical accuracy, clarity, and support for differential diagnosis.

Read article summary

Article

Added July 19, 2026

Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items

This innovation report presents an evaluated human-in-the-loop workflow for generating Concordance of Professional Judgment items from a locally defined ethics framework. ChatGPT-5 Thinking and Gemini 2.5 Pro each generated 28 Turkish items; after independent faculty review and adjudication, 25/28 and 27/28 items, respectively, were accepted or required only minor revision.

Read article summary

Article

Added July 5, 2026

Is AI the future of evaluation in medical education?? AI vs. human evaluation in objective structured clinical examination

This cross-sectional study compared OSCE scores from three human evaluators and two AI systems, ChatGPT-4o and Gemini Flash 1.5, for 196 pre-clinical medical students performing four clinical skills. AI scores were generally higher than human scores, with better alignment on visually observable steps than on auditory or communication-dependent criteria.

Read article summary

Fair-use

Problem-based learning scenario generation prompt

Added September 25, 2026

Prompt

Write a Problem-Based Learning (PBL) scenario by following the principles and structure below completely and exactly.
The scenario should be in the form of a text to be distributed to students; however, under each section there must be questions directed both to the students and to the tutor/facilitator.
The aim is for students to generate hypotheses, test these hypotheses, and formulate learning objectives that they will study and discuss in the next week's session. The learning objectives are to be derived by the students themselves; the facilitator supports this process of generating learning objectives, but does not provide information, rather guides the discussion.

1. GENERAL FRAMEWORK
The scenario must be student-centered.
The diagnosis must not be given at the beginning; information should progress gradually.
The scenario should be written in story format, chronological, realistic, fluent, and clinically accurate.
Each new piece of information introduced at every stage should be presented as a natural consequence of the information in the previous stage.
The aim is not memorization, but scientific reasoning by integrating basic and clinical knowledge.

2. SESSION STRUCTURE
The scenario must consist of 3 SESSIONS.
Session 1: Initial presentation. The information provided must NOT be sufficient to make any diagnosis clearly stand out. The first session should mainly suggest common causes.
Session 2: Further evaluation. The diagnosis should become understandable. It may go against the diagnosis that was prominent in the first part.
Session 3: Treatment, follow-up, and social/ethical soft skills dimension.
Each session must contain at least 3 and at most 4 sections.
Each session must be completable within a maximum of 2 hours.
Each session is attended by 6–8 students.

3. STORY AND FLOW
A realistic clinical and life history must be written.
The patient, family members, and physicians must be named.
Institutions must be explicitly stated.
The story must reflect the levels of the health system.

4. CLINICAL AND BASIC MEDICAL CONTENT
Physical examination, laboratory findings, and imaging should be added where appropriate. These data should not state the correct answer directly, whether diagnosis, treatment, mechanism, and so on, but should instead suggest the mechanism.

5. MANDATORY ELEMENTS UNDER EACH SECTION
A. STUDENT QUESTIONS
For each section, there must be 2 or 3 student questions appropriate to that section. The questions should be general, as in the examples. If necessary, the facilitator may deepen the discussion with additional questions.
Example question types:
A question asking students to summarize newly introduced information.
What are your hypotheses regarding this situation? Why?
What would you like to know from the patient's history? Why?
Which physical examination findings would you want to see? Why?
Which investigations would you request? Why?
The questions must NOT contain clues.
Do NOT provide examples in the questions.
Do NOT use parentheses.

B. GUIDING QUESTIONS FOR THE FACILITATOR
Under each section, there MUST also be 2–3 guiding questions for the facilitator. These are not given to students and are intended to help the facilitator manage the process.
These questions should help deepen students' thinking about underlying mechanisms and direct them toward learning objectives.
The questions should be written exactly as the facilitator might say them to the students, without giving clues.
Immediately below each facilitator question, there should be a one-sentence note stating the purpose of asking that question and which learning objective it is intended to address. This note is for the facilitator only.

6. LEARNING OBJECTIVES
Learning objectives must be included at the end of Sessions 1 and 2.
Learning objectives must definitely NOT be included in Session 3, because there will be no new session the following week. The process ends and everything becomes clear.
Session 3 may include social and ethical soft skills dimensions.
Learning objectives should be mechanism-focused rather than diagnosis-focused. They should be written using higher-order cognitive verbs such as explain, relate, interpret. They should be weighted toward basic medical sciences.
The learning objectives should be requested from the user of this prompt. If the user cannot provide them, you may write them yourself.

7. LANGUAGE AND STYLE
Academic, but close to the student
Clear, simple, and precise
Not in the tone of a lecturer explaining
Storytelling should be in the foreground and should arouse curiosity

8. REQUIRED OUTPUT FORMAT
Each PBL scenario must be presented in the following order:
Title, which should be intriguing and mysterious, possibly literary
Aim, which students will not see. This should aim not at memorizing the diagnosis, but at helping students learn to generate hypotheses, test these hypotheses, and understand the scientific reasoning process extending from the cell to the clinic.
Session and section numbers
Story text, with Session 1 Section 1 consisting of only 1–2 sentences
Student questions at the end of each section
Facilitator guiding questions at the end of each section, which students will not see
The learning objectives targeted in that session

Learning Objectives: [Learning Objectives]
Group Information: [(for example, second-year medical students)]

Source: Kucuk Bicer, B., Selim Kıyak, Y., & Coşkun, Ö. (2026). Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education. Interactive Learning Environments, 1-10.

Notes: The study used ChatGPT-5.2 Thinking in January 2026, generated each scenario in a separate conversation, and evaluated 10 AI scenarios matched to existing faculty scenarios.

Sources: Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education, Publisher article and supplementary material

Related subquestions: What can AI do for generating PBL scenarios?

Open prompt page

Creative Commons Attribution 4.0 International License (CC BY 4.0)

Concordance of Professional Judgment item generation prompt template

Added July 19, 2026

Prompt

Your role: You are an assessment designer creating Concordance-of-Professional-Judgment (CoPJ) items for health-profession learners. Items must reflect authentic ethical/professional dilemmas where more than one response can be defensible and expert opinions can diverge. Include expert explanations and a concluding key message.

Inputs (fill these before running):

• TOPIC: << e.g. reporting treatment errors; respect for confidentiality; breaking bad news; legal compliance; bedside >>.

• LEVEL: Undergraduate medical students.

• N_ITEMS: 1

• N_EXPERTS: <<e.g., 6–8; aim for several to show perspective diversity>>.

• LOCALE / POLICY HOOKS (optional): <Türk Tabipler Birliği Etik Bildirgeleri>.

• LANGUAGE: Turkish

Global requirements:

1. Vignette realism & brevity: Less than 50 words; clearly stated setting, role, and constraint; leave some ambiguity. Put the learner into the role. (e.g. “You are …”)

2. Response task: Present one specific behavior/attitude that a learner could take next. The learner will rate the behavior/attitude on a 4-point forced Likert scale:

Totally unacceptable | Unacceptable | Acceptable | Totally acceptable. Do not allow a neutral option.

3. Panel response: Provide an N_EXPERTS distribution across the four options for each behavior (e.g., 5/8 Totally unacceptable, 2/8 Unacceptable, 1/8 Acceptable).

4. Expert explanations: Include one short justification per expert (1-2 sentences each). Ensure credible disagreement across experts and make reasoning transparent.

5. Key message (educational synthesis): Conclude each item with a concise “Key message” as a narrative that includes:

o key principles,

o contextual nuance,

o cites relevant local policy/ethics codes, and

o offers 2–3 actionable “remember” bullets.

6. Tone & ethics: Respect patient dignity, colleagues, and uncertainty. Avoid implying a single “right answer”; emphasize reasoned judgment under uncertainty.

7. Length guardrails: Keep each item (vignette + behaviors + panel + explanations + key message) under ~450 words.

Structure to produce for the item:

1. Situation (vignette).

2. Behavior/attitude option (one), to be rated on the 4-point scale (no neutral).

3. Panel responses (counts out of N_EXPERTS per behavior), in bullet points.

4. Expert explanations (short, diverse justifications).

5. Key message (educator’s narrative synthesis with policy hooks).

Output Format (user-friendly prose layout):

Title: Concordance-of-Professional-Judgment Item

Topic: <Topic provided by the user>

Learning Objective: [A clear, one-sentence learning objective]

Vignette: [A short, context-inclusive, and unambiguous scenario of no more than 50 words]

Behaviour/Attitude: [A specific behaviour/attitude a learner could take next]

Question:

This behavior/attitude is:

☐ Totally unacceptable

☐ Unacceptable

☐ Acceptable

☐ Totally acceptable

Panel Feedback:

[List of 6–8 experts with name only with initials, role, Experience: years in practice, selected option, and justification]

Concordance Summary:

Totally unacceptable: x/x X%

Unacceptable: x/x X%

Acceptable: x/x X%

Totally acceptable: x/x X%

Educational Synthesis (as a narrative):

Key principle(s), contextual nuance, suggest consulting local policy / code of ethics, and finally offer 2–3 actionable “remember” bullets.

Tags: [relevant tags]

Quality checklist:

• Ambiguity present; multiple defensible positions; no single correct answer implied.

• Forced 4-point scale used; no neutral.

• Divergent expert opinions represented credibly.

• Explanations are transparent (cognitive apprenticeship feel).

• Key message cites or invites local policy/ethics when relevant (e.g., disclosure duties).

• Lengths within limits; respectful, professional tone.

Notes: Original full prompt evaluated with ChatGPT-5 Thinking and Gemini 2.5 Pro to generate CoPJ item drafts across 28 topics in the Turkish Medical Association's ethical declarations. Intended for formative content development with local parameter adaptation and expert review.

Sources: Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items, Supplementary Material 1 (full prompt)

Related subquestions: What can AI do for generating Concordance of Professional Judgment items?

Open prompt page