AI Script Concordance Test Generation in Medical Education

What can AI do for generating script concordance tests?

AI can produce rapid first drafts of Script Concordance Test (SCT) items when given a structured prompt, target learner level, clinical topic, and assessment focus. AI should not be treated as an autonomous SCT writer. The current evidence supports human-in-the-loop SCT development, not unsupervised use in summative or high-stakes assessment.

Evidence summary

Two studies evaluated LLM-generated SCTs with expert panels. In abdominal radiology, ChatGPT-4 and Claude 3 generated 10 SCT items; 16 radiologists judged most as assessing clinical reasoning, but only 73.33% of ChatGPT questions and 53.33% of Claude questions had acceptable response patterns. In obstetrics and gynecology, ChatGPT-4o and Claude 3.5 Sonnet generated 10 SCT items; 16 residents rated overall quality highly, with 90.57% and 91.48% agreement that items met criteria. The main weakness was matching difficulty to medical students.

Planning a study related to this question?

If you have a mature study idea, drafted protocol, or ethics-stage project related to this question or another gap in AI and medical education, you can share a collaboration proposal. Please include enough detail to assess fit, feasibility, and possible collaboration. Not every proposal will lead to collaboration.

Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License

Script Concordance Test generation prompt template in medical education

Prompt

You are a script concordance test (SCT) developer for medical exams for [GENERAL DESCRIPTION OF THE TARGET GROUP, SUCH AS, “undergraduate medical students”, “postgraduate medical trainees”, “continuing medical education participants”, OR A MORE DETAILED INFORMATION, SUCH AS “last year radiology residents in Turkey”].

SCT is a method used to assess clinical reasoning and decision-making skills in healthcare stu-dents and professionals. It is designed to evaluate how they interpret clinical information and make decisions under conditions of uncertainty. Each scenario is a clinical vignette describing a medical situation. For each scenario, there are multiple possible options or actions that a physician could take or consider in that situation.

Each SCT consists of four main components.

1. A very brief clinical vignette on a clinical problem regarding [TYPE WHAT THE SET OF QUESTIONS SHOULD FOCUS ON: E.G. diagnosis, order of investigation, treatment].

In a single table with four rows, the first row includes the labels:
2. First column: ONLY three different key plausible [TYPE WHAT THE SET OF QUESTIONS SHOULD FOCUS ON: E.G. diagnosis, order of investigation, treatment] for the clinical vignette. The label of the column should be “If you were thinking of …”.
3. Second column: A new piece of clinical information for each of the initial option to make the situation significantly more complex based on the clinical vignette, such as, new symptoms, previously undisclosed aspects of the patient’s history, physical examination findings, and results of tests or previous treatments. The label of the column should be “Then you learn that …”.
4. Third column: Asking the examinee to rate on a 5-point scale (from -2 to +2) how this new information impacts or affects the initial option - whether it makes that option more or less appro-priate, and to what degree. The label of the column should be “It becomes … (-2 much less appro-priate, -1 less appropriate, 0 no change, +1 more appropriate, +2 much more appropriate)”.

So the examinee has to reason through how the new clinical data changes or updates the appropriateness of the initial [TYPE WHAT THE SET OF QUESTIONS SHOULD FOCUS ON: E.G. diagnosis, order of investigation, treatment] option they were given. The 5-point scale allows them to indicate if the new information makes the original option more likely, less likely, or doesn't really change it, and to what extent. The idea is that this tests the student's clinical reasoning abilities by presenting evolving, uncertain medical cases and having them re-evaluate options as new information comes in, which is what happens in real medical practice.

Write a SCT for the following clinical problem: [TYPE A CLINICAL PROBLEM OR TOPIC OR LEARNING OBJECTIVE RELATED TO DIAGNOSIS, INVESTIGATION, OR TREATMENT] according to [TYPE THE NAME OF A RELEVANT GUIDELINE OR JUST TYPE “the reliable guidelines and/or expert consensus on this clinical problem”]. Indicate the level of appropriateness in the third column cells, together with a very brief rationale.

Source: Kıyak, Y.S., & Emekli, E. (2024). A prompt for generating script concordance test using ChatGPT, Claude, and llama large language model Chatbots. Spanish Journal of Medical Education, 5(3).

Sources: Original source

Related subquestions: What can AI do for generating script concordance tests?

Open prompt page

What can AI do for generating script concordance tests?

Evidence summary

Practical starting points

Planning a study related to this question?

Selected studies

Large language models for generating script concordance test in obstetrics and gynecology: ChatGPT and Claude

Using Large Language Models to Generate Script Concordance Test in Medical Education: ChatGPT and Claude

Related prompt architectures

Script Concordance Test generation prompt template in medical education

Linked guides

How to review AI-generated script concordance tests?