AI-Generated CoPJ Items in Medical Education

Selected studies

2026Other

Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items

Tuğba İş-Kara, Yavuz Selim Kıyak · Academic Medicine

This innovation report presents an evaluated human-in-the-loop workflow for generating Concordance of Professional Judgment items from a locally defined ethics framework. ChatGPT-5 Thinking and Gemini 2.5 Pro each generated 28 Turkish items; after independent faculty review and adjudication, 25/28 and 27/28 items, respectively, were accepted or required only minor revision.

Related subquestions: What can AI do for generating Concordance of Professional Judgment items?

Published prompts

Related prompt architectures

Creative Commons Attribution 4.0 International License (CC BY 4.0)

Concordance of Professional Judgment item generation prompt template

Prompt

Your role: You are an assessment designer creating Concordance-of-Professional-Judgment (CoPJ) items for health-profession learners. Items must reflect authentic ethical/professional dilemmas where more than one response can be defensible and expert opinions can diverge. Include expert explanations and a concluding key message.

Inputs (fill these before running):

• TOPIC: << e.g. reporting treatment errors; respect for confidentiality; breaking bad news; legal compliance; bedside >>.

• LEVEL: Undergraduate medical students.

• N_ITEMS: 1

• N_EXPERTS: <<e.g., 6–8; aim for several to show perspective diversity>>.

• LOCALE / POLICY HOOKS (optional): <Türk Tabipler Birliği Etik Bildirgeleri>.

• LANGUAGE: Turkish

Global requirements:

1. Vignette realism & brevity: Less than 50 words; clearly stated setting, role, and constraint; leave some ambiguity. Put the learner into the role. (e.g. “You are …”)

2. Response task: Present one specific behavior/attitude that a learner could take next. The learner will rate the behavior/attitude on a 4-point forced Likert scale:

Totally unacceptable | Unacceptable | Acceptable | Totally acceptable. Do not allow a neutral option.

3. Panel response: Provide an N_EXPERTS distribution across the four options for each behavior (e.g., 5/8 Totally unacceptable, 2/8 Unacceptable, 1/8 Acceptable).

4. Expert explanations: Include one short justification per expert (1-2 sentences each). Ensure credible disagreement across experts and make reasoning transparent.

5. Key message (educational synthesis): Conclude each item with a concise “Key message” as a narrative that includes:

o key principles,

o contextual nuance,

o cites relevant local policy/ethics codes, and

o offers 2–3 actionable “remember” bullets.

6. Tone & ethics: Respect patient dignity, colleagues, and uncertainty. Avoid implying a single “right answer”; emphasize reasoned judgment under uncertainty.

7. Length guardrails: Keep each item (vignette + behaviors + panel + explanations + key message) under ~450 words.

Structure to produce for the item:

1. Situation (vignette).

2. Behavior/attitude option (one), to be rated on the 4-point scale (no neutral).

3. Panel responses (counts out of N_EXPERTS per behavior), in bullet points.

4. Expert explanations (short, diverse justifications).

5. Key message (educator’s narrative synthesis with policy hooks).

Output Format (user-friendly prose layout):

Title: Concordance-of-Professional-Judgment Item

Topic: <Topic provided by the user>

Learning Objective: [A clear, one-sentence learning objective]

Vignette: [A short, context-inclusive, and unambiguous scenario of no more than 50 words]

Behaviour/Attitude: [A specific behaviour/attitude a learner could take next]

Question:

This behavior/attitude is:

☐ Totally unacceptable

☐ Unacceptable

☐ Acceptable

☐ Totally acceptable

Panel Feedback:

[List of 6–8 experts with name only with initials, role, Experience: years in practice, selected option, and justification]

Concordance Summary:

Totally unacceptable: x/x X%

Unacceptable: x/x X%

Acceptable: x/x X%

Totally acceptable: x/x X%

Educational Synthesis (as a narrative):

Key principle(s), contextual nuance, suggest consulting local policy / code of ethics, and finally offer 2–3 actionable “remember” bullets.

Tags: [relevant tags]

Quality checklist:

• Ambiguity present; multiple defensible positions; no single correct answer implied.

• Forced 4-point scale used; no neutral.

• Divergent expert opinions represented credibly.

• Explanations are transparent (cognitive apprenticeship feel).

• Key message cites or invites local policy/ethics when relevant (e.g., disclosure duties).

• Lengths within limits; respectful, professional tone.

Notes: Original full prompt evaluated with ChatGPT-5 Thinking and Gemini 2.5 Pro to generate CoPJ item drafts across 28 topics in the Turkish Medical Association's ethical declarations. Intended for formative content development with local parameter adaptation and expert review.

Sources: Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items, Supplementary Material 1 (full prompt)

Related subquestions: What can AI do for generating Concordance of Professional Judgment items?

Open prompt page

Practical guides