Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education
Kucuk Bicer B, Kıyak YS, Coşkun Ö. Expert evaluation of AI-assisted scenario design for interactive problem based learning environments in medical education. Interactive Learning Environments. Published online September 18, 2026. doi:10.1080/10494820.2026.2734312.
Abstract or concise summary
This single-institution paired comparative study evaluated 10 matched pairs of existing faculty-authored and ChatGPT-generated PBL scenarios addressing identical learning objectives. Twenty-two experienced academic PBL educators, blinded to source, rated the scenarios with an author-developed 24-item checklist. Across valid ratings, AI-generated scenarios received 90.7% positive responses compared with 80.3% for human-written scenarios. The largest descriptive advantages favored AI-generated scenarios for cognitive load balance, consistency, scientific or clinical accuracy, clarity, and differential diagnosis support, while human scenarios were rated slightly higher for psychosocial context and cultural sensitivity. The study evaluates expert-rated scenario quality, not learner outcomes.
DOI: 10.1080/10494820.2026.2734312
URL: https://doi.org/10.1080/10494820.2026.2734312
Journal: Interactive Learning Environments
Authors: Burcu Kucuk Bicer, Yavuz Selim Kıyak, Özlem Coşkun
Last updated: September 20, 2026
Article interpretation
- Main contribution
- This matched, source-blinded expert evaluation found that PBL scenarios generated from the same learning objectives with a structured one-shot prompt received a higher overall proportion of positive checklist ratings than existing faculty-authored scenarios: 90.7% versus 80.3%. The strongest descriptive differences favored the AI scenarios for cognitive load balance, consistency, scientific or clinical accuracy, clarity, and support for differential diagnosis.
- Practical use
- Educators can use the study's prompt to draft progressively disclosed PBL scenarios from explicit learning objectives and a defined learner group. Generated drafts should undergo expert review for clinical accuracy, alignment, cognitive load, clueing, facilitator questions, learner level, psychosocial content, cultural fit, and local health-system context before use. Learner piloting is needed before broader implementation.
- Main caution
- The study compared existing faculty scenarios developed through routine institutional processes with AI scenarios produced under a detailed evidence-informed scaffold. It therefore does not isolate AI authorship from differences in authoring workflow. The evidence comes from one institution, 10 pairs, two to three raters per pair, an author-developed three-category checklist, and expert judgments rather than learner outcomes.
- Research gap
- Future studies should compare AI and human authors under equivalent design instructions, test scenarios with learners, measure engagement and learning outcomes, quantify faculty editing time and revision burden, validate the review instrument independently, and replicate across institutions, cultures, specialties, models, and prompt versions.
Collaboration
Working on a study that could address this gap?
If you have a mature study idea, drafted protocol, or ethics-stage project related to this research gap or another gap in AI and medical education, you can share a collaboration proposal. Please include enough detail to assess fit, feasibility, and possible collaboration. Not every proposal will lead to collaboration.
Curator note
The most defensible interpretation is that a constrained human-in-the-loop workflow can produce structurally strong PBL drafts.
Caution
Prompt / method
- Includes prompt
- Yes
Related links
Related main question areas: What can AI do for generating scenarios?
Related subquestions: What can AI do for generating PBL scenarios?
Related published prompts: Problem-based learning scenario generation prompt
Tags: