How to Review AI-Generated Key-Feature Questions

Copyable resource

Resource

Kiyak’s quality checklist for evaluating Script Concordance Test

a.	The scenario described a challenging situation, even for experts.
b.	The scenario was of appropriate difficulty for medical students.
c.	Reading the scenario was necessary to understand the question and context.
d.	The clinical presentation in the scenario was typical.
e.	The scenario was appropriately written.
f.	The questions (presented in sets of three options) were prepared to include important aspects related to the topic.
g.	Options were relevant to the case.
h.	The same option did not appear in two consecutive questions.
i.	It was possible to test the relationship between the added new information (2nd column) and the initially presented option (1st column).
j.	The Likert scale (3rd column) was clearly and explicitly defined.
k.	The questions were designed to distribute answers evenly across all values of the Likert scale.
l.	The questions allowed for balanced ambiguity, enabling different interpretations by experts.

Source: Kıyak, Y. S., & Emekli, E. (2024). Using Large Language Models to Generate Script Concordance Test in Medical Education: ChatGPT and Claude. Spanish Journal of Medical Education, 6(1). https://doi.org/10.6018/edumed.636331

Related evidence

Linked articles

2025

Large language models for generating script concordance test in obstetrics and gynecology: ChatGPT and Claude

Zuhal Yapıcı Coşkun, Yavuz Selim Kıyak, Özlem Coşkun, Işıl İrem Budakoğlu, Özhan Özdemir · Medical Teacher

This cross-sectional study evaluated ChatGPT-4o and Claude 3.5 Sonnet for generating obstetrics and gynecology SCT items on five primary-care diagnostic topics. Sixteen obstetrics and gynecology residents rated 10 AI-generated SCT items against 11 criteria; overall agreement that items met criteria was 90.57% for ChatGPT-4o and 91.48% for Claude 3.5 Sonnet.

2024Other

Using Large Language Models to Generate Script Concordance Test in Medical Education: ChatGPT and Claude

Yavuz Selim Kıyak, Emre Emekli · Revista Española de Educación Médica

This study tested whether ChatGPT-4 and Claude 3 Sonnet could generate Script Concordance Test items for abdominal radiology using a detailed prompt. Sixteen radiologists judged the items; most rated them as assessing clinical reasoning rather than factual recall, and ChatGPT-4 had higher acceptability than Claude across the reported indicators.

Related published prompts

Linked prompt architectures

Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License

Script Concordance Test generation prompt template in medical education

Prompt

You are a script concordance test (SCT) developer for medical exams for [GENERAL DESCRIPTION OF THE TARGET GROUP, SUCH AS, “undergraduate medical students”, “postgraduate medical trainees”, “continuing medical education participants”, OR A MORE DETAILED INFORMATION, SUCH AS “last year radiology residents in Turkey”].

SCT is a method used to assess clinical reasoning and decision-making skills in healthcare stu-dents and professionals. It is designed to evaluate how they interpret clinical information and make decisions under conditions of uncertainty. Each scenario is a clinical vignette describing a medical situation. For each scenario, there are multiple possible options or actions that a physician could take or consider in that situation.

Each SCT consists of four main components.

1. A very brief clinical vignette on a clinical problem regarding [TYPE WHAT THE SET OF QUESTIONS SHOULD FOCUS ON: E.G. diagnosis, order of investigation, treatment].

In a single table with four rows, the first row includes the labels:
2. First column: ONLY three different key plausible [TYPE WHAT THE SET OF QUESTIONS SHOULD FOCUS ON: E.G. diagnosis, order of investigation, treatment] for the clinical vignette. The label of the column should be “If you were thinking of …”.
3. Second column: A new piece of clinical information for each of the initial option to make the situation significantly more complex based on the clinical vignette, such as, new symptoms, previously undisclosed aspects of the patient’s history, physical examination findings, and results of tests or previous treatments. The label of the column should be “Then you learn that …”.
4. Third column: Asking the examinee to rate on a 5-point scale (from -2 to +2) how this new information impacts or affects the initial option - whether it makes that option more or less appro-priate, and to what degree. The label of the column should be “It becomes … (-2 much less appro-priate, -1 less appropriate, 0 no change, +1 more appropriate, +2 much more appropriate)”.

So the examinee has to reason through how the new clinical data changes or updates the appropriateness of the initial [TYPE WHAT THE SET OF QUESTIONS SHOULD FOCUS ON: E.G. diagnosis, order of investigation, treatment] option they were given. The 5-point scale allows them to indicate if the new information makes the original option more likely, less likely, or doesn't really change it, and to what extent. The idea is that this tests the student's clinical reasoning abilities by presenting evolving, uncertain medical cases and having them re-evaluate options as new information comes in, which is what happens in real medical practice.

Write a SCT for the following clinical problem: [TYPE A CLINICAL PROBLEM OR TOPIC OR LEARNING OBJECTIVE RELATED TO DIAGNOSIS, INVESTIGATION, OR TREATMENT] according to [TYPE THE NAME OF A RELEVANT GUIDELINE OR JUST TYPE “the reliable guidelines and/or expert consensus on this clinical problem”]. Indicate the level of appropriateness in the third column cells, together with a very brief rationale.

Source: Kıyak, Y.S., & Emekli, E. (2024). A prompt for generating script concordance test using ChatGPT, Claude, and llama large language model Chatbots. Spanish Journal of Medical Education, 5(3).

Sources: Original source

Related subquestions: What can AI do for generating script concordance tests?

Open prompt page

How to review AI-generated script concordance tests?