How to Review AI-Generated CoPJ Items

How to review AI-generated Concordance of Professional Judgment items?

Use the study's 20-criterion checklist to judge whether an AI-generated CoPJ item is ready to accept, revise, or reject before formative use.

Purpose: Use the study's 20-criterion checklist to judge whether an AI-generated CoPJ item is ready to accept, revise, or reject before formative use.

Source: Original article, Table 1

Copyright / reuse status: Creative Commons Attribution 4.0 International License (CC BY 4.0)

Copyable resource

20-criterion checklist for reviewing AI-generated CoPJ items

Rate each criterion No (0) or Yes (1), resolve disagreements by adjudication, and then record a separate global educational-suitability judgment.

Resource

20-criterion checklist for reviewing AI-generated Concordance of Professional Judgment (CoPJ) items

Instructions

Rate every criterion as No (0) or Yes (1). If reviewers disagree, reexamine the item and adjudicate the criterion. After completing the checklist, record a separate global educational-suitability judgment.

A. Vignette and response options

1. The vignette presents a realistic situation.  [ ] No (0)  [ ] Yes (1)
2. The vignette is concise yet specifies the necessary elements.  [ ] No (0)  [ ] Yes (1)
3. The vignette aligns with the intended theme.  [ ] No (0)  [ ] Yes (1)
4. Uncertainty is present to prompt judgment rather than fact recall.  [ ] No (0)  [ ] Yes (1)
5. A plausible next action is presented as a behavior or attitude.  [ ] No (0)  [ ] Yes (1)
6. The vignette puts the candidate into the role (eg, “You are…” rather than “A patient presents…”).  [ ] No (0)  [ ] Yes (1)
7. Reading the vignette is necessary to understand the next action.  [ ] No (0)  [ ] Yes (1)

B. Response format (scale)

8. A 4-point Likert scale without a neutral point is present.  [ ] No (0)  [ ] Yes (1)

C. Expert panel

9. There are comments from 6 to 8 experts.  [ ] No (0)  [ ] Yes (1)
10. The experts have diverse backgrounds.  [ ] No (0)  [ ] Yes (1)
11. There is no factual inaccuracy in the expert comments.  [ ] No (0)  [ ] Yes (1)
12. Expert responses are diverse, including both positive and negative values.  [ ] No (0)  [ ] Yes (1)
13. Explanations serve as worked examples that make tacit expertise visible.  [ ] No (0)  [ ] Yes (1)
14. Each expert gives a concise rationale.  [ ] No (0)  [ ] Yes (1)

D. Concordance summary

15. The values (numbers and percentages) in the concordance summary are correct.  [ ] No (0)  [ ] Yes (1)

E. Synthesis (ie, key message)

16. A concise synthesis highlights key messages.  [ ] No (0)  [ ] Yes (1)
17. There is no factual inaccuracy in the synthesis.  [ ] No (0)  [ ] Yes (1)
18. The synthesis connects to relevant codes and/or guidelines.  [ ] No (0)  [ ] Yes (1)

F. General evaluation

19. The ambiguity level is appropriate for the learner level.  [ ] No (0)  [ ] Yes (1)
20. The complexity is appropriate for the learner level.  [ ] No (0)  [ ] Yes (1)

Checklist score: ____ / 20

Global educational-suitability judgment

[ ] Accepted
[ ] Minor revision
[ ] Major revision
[ ] Rejected

Reviewer notes:


Source: İş-Kara T, Kıyak YS. Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items. Academic Medicine. Published online May 9, 2026. doi:10.1093/acamed/wvag147. Checklist criteria reproduced from Table 1 under CC BY 4.0.

Related evidence

Linked articles

2026Other

Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items

Tuğba İş-Kara, Yavuz Selim Kıyak · Academic Medicine

This innovation report presents an evaluated human-in-the-loop workflow for generating Concordance of Professional Judgment items from a locally defined ethics framework. ChatGPT-5 Thinking and Gemini 2.5 Pro each generated 28 Turkish items; after independent faculty review and adjudication, 25/28 and 27/28 items, respectively, were accepted or required only minor revision.

Related published prompts

Linked prompt architectures

Creative Commons Attribution 4.0 International License (CC BY 4.0)

Concordance of Professional Judgment item generation prompt template

Prompt

Your role: You are an assessment designer creating Concordance-of-Professional-Judgment (CoPJ) items for health-profession learners. Items must reflect authentic ethical/professional dilemmas where more than one response can be defensible and expert opinions can diverge. Include expert explanations and a concluding key message.

Inputs (fill these before running):

• TOPIC: << e.g. reporting treatment errors; respect for confidentiality; breaking bad news; legal compliance; bedside >>.

• LEVEL: Undergraduate medical students.

• N_ITEMS: 1

• N_EXPERTS: <<e.g., 6–8; aim for several to show perspective diversity>>.

• LOCALE / POLICY HOOKS (optional): <Türk Tabipler Birliği Etik Bildirgeleri>.

• LANGUAGE: Turkish

Global requirements:

1. Vignette realism & brevity: Less than 50 words; clearly stated setting, role, and constraint; leave some ambiguity. Put the learner into the role. (e.g. “You are …”)

2. Response task: Present one specific behavior/attitude that a learner could take next. The learner will rate the behavior/attitude on a 4-point forced Likert scale:

Totally unacceptable | Unacceptable | Acceptable | Totally acceptable. Do not allow a neutral option.

3. Panel response: Provide an N_EXPERTS distribution across the four options for each behavior (e.g., 5/8 Totally unacceptable, 2/8 Unacceptable, 1/8 Acceptable).

4. Expert explanations: Include one short justification per expert (1-2 sentences each). Ensure credible disagreement across experts and make reasoning transparent.

5. Key message (educational synthesis): Conclude each item with a concise “Key message” as a narrative that includes:

o key principles,

o contextual nuance,

o cites relevant local policy/ethics codes, and

o offers 2–3 actionable “remember” bullets.

6. Tone & ethics: Respect patient dignity, colleagues, and uncertainty. Avoid implying a single “right answer”; emphasize reasoned judgment under uncertainty.

7. Length guardrails: Keep each item (vignette + behaviors + panel + explanations + key message) under ~450 words.

Structure to produce for the item:

1. Situation (vignette).

2. Behavior/attitude option (one), to be rated on the 4-point scale (no neutral).

3. Panel responses (counts out of N_EXPERTS per behavior), in bullet points.

4. Expert explanations (short, diverse justifications).

5. Key message (educator’s narrative synthesis with policy hooks).

Output Format (user-friendly prose layout):

Title: Concordance-of-Professional-Judgment Item

Topic: <Topic provided by the user>

Learning Objective: [A clear, one-sentence learning objective]

Vignette: [A short, context-inclusive, and unambiguous scenario of no more than 50 words]

Behaviour/Attitude: [A specific behaviour/attitude a learner could take next]

Question:

This behavior/attitude is:

☐ Totally unacceptable

☐ Unacceptable

☐ Acceptable

☐ Totally acceptable

Panel Feedback:

[List of 6–8 experts with name only with initials, role, Experience: years in practice, selected option, and justification]

Concordance Summary:

Totally unacceptable: x/x X%

Unacceptable: x/x X%

Acceptable: x/x X%

Totally acceptable: x/x X%

Educational Synthesis (as a narrative):

Key principle(s), contextual nuance, suggest consulting local policy / code of ethics, and finally offer 2–3 actionable “remember” bullets.

Tags: [relevant tags]

Quality checklist:

• Ambiguity present; multiple defensible positions; no single correct answer implied.

• Forced 4-point scale used; no neutral.

• Divergent expert opinions represented credibly.

• Explanations are transparent (cognitive apprenticeship feel).

• Key message cites or invites local policy/ethics when relevant (e.g., disclosure duties).

• Lengths within limits; respectful, professional tone.

Notes: Original full prompt evaluated with ChatGPT-5 Thinking and Gemini 2.5 Pro to generate CoPJ item drafts across 28 topics in the Turkish Medical Association's ethical declarations. Intended for formative content development with local parameter adaptation and expert review.

Sources: Large language models in professionalism and ethical decision-making education: a transferable process for generating concordance of professional judgment items, Supplementary Material 1 (full prompt)

Related subquestions: What can AI do for generating Concordance of Professional Judgment items?

Open prompt page