How to review AI-generated PBL scenarios?

Use the study's 24-item expert checklist to review an AI-generated PBL scenario before learner use.

Purpose: Use the study's 24-item expert checklist to review an AI-generated PBL scenario before learner use.

Source: Supplementary Table S1

Copyright / reuse status: Fair-use

Checklist

Source: Kucuk Bicer, B., Selim Kıyak, Y., & Coşkun, Ö. (2026). Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education. Interactive Learning Environments, 1-10.
1. Clear Learning Objectives: Are there defined “Intended Learning Objectives” for the faculty?
2. Alignment: Does the scenario direct students to the intended learning objectives?
3. Feasibility: Is the scope of the learning objectives realistic for the time functioning (three separate weeks, 2 hours of discussion by 6-8 student in each session)?
4. Breadth: Does the scenario prompt study in multiple medical domains (Basic Science, Clinical Medicine)?
5. Psychosocial context: Does the scenario prompt considering any psychosocial aspect such as communication, professionalism, ethics?
6. Authenticity: Is the scenario realistic?
7. Consistency: Is the scenario consistent?
8. Relevance: Is the presenting complaint common or critical enough that a generalist doctor needs to know it?
9. Ambiguity Level: Is the problem messy enough, not immediately solvable, requiring research to understand?
10. Progressive Disclosure: Is the information broken down into coherent stages?
11. The Hook: Does the very first paragraph provoke immediate wide range of hypotheses?
12. Anonymity: Is the patient name fictional?
13. Clarity: Is the language clear?
14. Data Accuracy: Is the provided information scientifically/clinically correct?
15. Prompt Questions: Are there specific questions listed to help the facilitator nudge students?
16. Target Learner Appropriateness: Is the scenario appropriate for the intended learner level.
17. Cognitive Load: Is the complexity of the scenario balanced to avoid cognitive overload while still providing sufficient challenge?
18. Learning Objective Generation: Does the scenario encourage students to identify their own learning objective?
19. Knowledge Gaps: Does the case naturally reveal gaps in students' knowledge (What do we need to learn?)?
20. Evidence-Based Inquiry: Does the scenario prompt students to seek evidence-based resources?
21. Hypothesis Generation: Does the scenario require students to generate and revise multiple hypotheses?
22. Differential Diagnosis: Does the scenario support development of differential diagnosis skills?
23. Health Care System Relevance: Is the scenario appropriate for the local health care system (e.g.primary care, referral pathways)?
24. Cultural Sensitivity: Does the scenario reflect culturally appropriate patient-physician interactions?

Copyable resource

24-item checklist for reviewing a PBL scenario

Rate each criterion as Yes, No, or No idea.

Resource

PBL Evaluator Guidance

In this study, we aim to compare learning scenarios developed by human educators with those generated using artificial intelligence tools. Your role as an evaluator is to review these scenarios based on their educational quality, structure, and relevance, rather than on who or what created them.

Evaluators are expected to assess each domain holistically rather than as isolated checklist items. While reviewing each scenario, please focus on how well it supports learning. Consider whether the objectives are clear, whether the scenario is realistic and clinically meaningful, and whether it encourages learners to think, discuss, and explore the problem rather than immediately reaching an answer. The goal is to understand how effectively each scenario promotes understanding, clinical reasoning, and self-directed learning.

It is important to evaluate the scenarios as they are presented, without trying to guess their origin. We are interested in the strengths and limitations of the scenarios themselves, not in preconceived ideas about human or AI-generated content. There are no right or wrong answers in this evaluation; your professional judgment and experience are the most valuable components of this process.

Please use the provided criteria to guide your evaluation, but feel free to consider the overall educational value of each scenario as a whole. Your feedback will contribute directly to improving scenario design and to a better understanding of how artificial intelligence can be used responsibly and effectively in medical education.

Thank you very much for your time, expertise, and contribution to this study. Your participation is greatly appreciated.

On Behalf of the Study Group

Evaluation Process

For each item, please indicate the option that best reflects your judgment.

1: Yes- The criterion is clearly met.
2: No- The criterion is not met.
3: No idea- There is insufficient information in the scenario to make a judgment, or the criterion is unclear.

*Please base your responses solely on the content of the scenario provided. If you are uncertain or the information is not explicitly or implicitly addressed, select No idea.

Evaluation Checklist

1. Clear Learning Objectives: Are there defined “Intended Learning Objectives” for the faculty?
2. Alignment: Does the scenario direct students to the intended learning objectives?
3. Feasibility: Is the scope of the learning objectives realistic for the time functioning (three separate weeks, 2 hours of discussion by 6-8 student in each session)?
4. Breadth: Does the scenario prompt study in multiple medical domains (Basic Science, Clinical Medicine)?
5. Psychosocial context: Does the scenario prompt considering any psychosocial aspect such as communication, professionalism, ethics?
6. Authenticity: Is the scenario realistic?
7. Consistency: Is the scenario consistent?
8. Relevance: Is the presenting complaint common or critical enough that a generalist doctor needs to know it?
9. Ambiguity Level: Is the problem messy enough, not immediately solvable, requiring research to understand?
10. Progressive Disclosure: Is the information broken down into coherent stages?
11. The Hook: Does the very first paragraph provoke immediate wide range of hypotheses?
12. Anonymity: Is the patient name fictional?
13. Clarity: Is the language clear?
14. Data Accuracy: Is the provided information scientifically/clinically correct?
15. Prompt Questions: Are there specific questions listed to help the facilitator nudge students?
16. Target Learner Appropriateness: Is the scenario appropriate for the intended learner level.
17. Cognitive Load: Is the complexity of the scenario balanced to avoid cognitive overload while still providing sufficient challenge?
18. Learning Objective Generation: Does the scenario encourage students to identify their own learning objective?
19. Knowledge Gaps: Does the case naturally reveal gaps in students' knowledge (What do we need to learn?)?
20. Evidence-Based Inquiry: Does the scenario prompt students to seek evidence-based resources?
21. Hypothesis Generation: Does the scenario require students to generate and revise multiple hypotheses?
22. Differential Diagnosis: Does the scenario support development of differential diagnosis skills?
23. Health Care System Relevance: Is the scenario appropriate for the local health care system (e.g.primary care, referral pathways)?
24. Cultural Sensitivity: Does the scenario reflect culturally appropriate patient-physician interactions?

Related evidence

Linked articles

2026Non-randomized experiment

Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education

Burcu Kucuk Bicer, Yavuz Selim Kıyak, Özlem Coşkun · Interactive Learning Environments

This matched, source-blinded expert evaluation found that PBL scenarios generated from the same learning objectives with a structured one-shot prompt received a higher overall proportion of positive checklist ratings than existing faculty-authored scenarios: 90.7% versus 80.3%. The strongest descriptive differences favored the AI scenarios for cognitive load balance, consistency, scientific or clinical accuracy, clarity, and support for differential diagnosis.

Related published prompts

Linked prompt architectures

Fair-use

Problem-based learning scenario generation prompt

Prompt

Write a Problem-Based Learning (PBL) scenario by following the principles and structure below completely and exactly.
The scenario should be in the form of a text to be distributed to students; however, under each section there must be questions directed both to the students and to the tutor/facilitator.
The aim is for students to generate hypotheses, test these hypotheses, and formulate learning objectives that they will study and discuss in the next week's session. The learning objectives are to be derived by the students themselves; the facilitator supports this process of generating learning objectives, but does not provide information, rather guides the discussion.

1. GENERAL FRAMEWORK
The scenario must be student-centered.
The diagnosis must not be given at the beginning; information should progress gradually.
The scenario should be written in story format, chronological, realistic, fluent, and clinically accurate.
Each new piece of information introduced at every stage should be presented as a natural consequence of the information in the previous stage.
The aim is not memorization, but scientific reasoning by integrating basic and clinical knowledge.

2. SESSION STRUCTURE
The scenario must consist of 3 SESSIONS.
Session 1: Initial presentation. The information provided must NOT be sufficient to make any diagnosis clearly stand out. The first session should mainly suggest common causes.
Session 2: Further evaluation. The diagnosis should become understandable. It may go against the diagnosis that was prominent in the first part.
Session 3: Treatment, follow-up, and social/ethical soft skills dimension.
Each session must contain at least 3 and at most 4 sections.
Each session must be completable within a maximum of 2 hours.
Each session is attended by 6–8 students.

3. STORY AND FLOW
A realistic clinical and life history must be written.
The patient, family members, and physicians must be named.
Institutions must be explicitly stated.
The story must reflect the levels of the health system.

4. CLINICAL AND BASIC MEDICAL CONTENT
Physical examination, laboratory findings, and imaging should be added where appropriate. These data should not state the correct answer directly, whether diagnosis, treatment, mechanism, and so on, but should instead suggest the mechanism.

5. MANDATORY ELEMENTS UNDER EACH SECTION
A. STUDENT QUESTIONS
For each section, there must be 2 or 3 student questions appropriate to that section. The questions should be general, as in the examples. If necessary, the facilitator may deepen the discussion with additional questions.
Example question types:
A question asking students to summarize newly introduced information.
What are your hypotheses regarding this situation? Why?
What would you like to know from the patient's history? Why?
Which physical examination findings would you want to see? Why?
Which investigations would you request? Why?
The questions must NOT contain clues.
Do NOT provide examples in the questions.
Do NOT use parentheses.

B. GUIDING QUESTIONS FOR THE FACILITATOR
Under each section, there MUST also be 2–3 guiding questions for the facilitator. These are not given to students and are intended to help the facilitator manage the process.
These questions should help deepen students' thinking about underlying mechanisms and direct them toward learning objectives.
The questions should be written exactly as the facilitator might say them to the students, without giving clues.
Immediately below each facilitator question, there should be a one-sentence note stating the purpose of asking that question and which learning objective it is intended to address. This note is for the facilitator only.

6. LEARNING OBJECTIVES
Learning objectives must be included at the end of Sessions 1 and 2.
Learning objectives must definitely NOT be included in Session 3, because there will be no new session the following week. The process ends and everything becomes clear.
Session 3 may include social and ethical soft skills dimensions.
Learning objectives should be mechanism-focused rather than diagnosis-focused. They should be written using higher-order cognitive verbs such as explain, relate, interpret. They should be weighted toward basic medical sciences.
The learning objectives should be requested from the user of this prompt. If the user cannot provide them, you may write them yourself.

7. LANGUAGE AND STYLE
Academic, but close to the student
Clear, simple, and precise
Not in the tone of a lecturer explaining
Storytelling should be in the foreground and should arouse curiosity

8. REQUIRED OUTPUT FORMAT
Each PBL scenario must be presented in the following order:
Title, which should be intriguing and mysterious, possibly literary
Aim, which students will not see. This should aim not at memorizing the diagnosis, but at helping students learn to generate hypotheses, test these hypotheses, and understand the scientific reasoning process extending from the cell to the clinic.
Session and section numbers
Story text, with Session 1 Section 1 consisting of only 1–2 sentences
Student questions at the end of each section
Facilitator guiding questions at the end of each section, which students will not see
The learning objectives targeted in that session

Learning Objectives: [Learning Objectives]
Group Information: [(for example, second-year medical students)]

Source: Kucuk Bicer, B., Selim Kıyak, Y., & Coşkun, Ö. (2026). Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education. Interactive Learning Environments, 1-10.

Notes: The study used ChatGPT-5.2 Thinking in January 2026, generated each scenario in a separate conversation, and evaluated 10 AI scenarios matched to existing faculty scenarios.

Sources: Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education, Publisher article and supplementary material

Related subquestions: What can AI do for generating PBL scenarios?

Open prompt page