Automated scoring of student videos in medical education: a comparison between a large language model and expert evaluation

Abstract or concise summary

Video-based assignments are used in medical education, yet expert scoring is time-intensive. Large language models (LLMs) offer scalable alternatives, but their validity for evaluating multimodal student work is uncertain. We examined whether a state-of-the-art LLM (Gemini 2.5 Pro) could approximate expert scoring of medical student video presentations in evidence-based medicine. A total of 139 student submissions were evaluated by an experienced faculty member (reference standard) and by the LLM under two prompting strategies: (i) rubric-only and (ii) critical expert-style. Scores across 12 rubric items (maximum 95 points) were compared using paired t -tests, effect sizes, and Bland-Altman analysis. Expert scoring yielded a mean total of 62.0 (SD 13.6). The rubric-only prompt systematically overestimated performance (mean 87.7, SD 8.0, P < 0.001; bias –25.7). The critical prompt produced lower scores (mean 53.5, SD 10.4, P < 0.001; bias +8.5). At the item level, rubric-only prompting aligned better with mechanical tasks (e.g., keywords and referencing), whereas the critical prompt penalized appraisal and synthesis disproportionately. Prompting strategy substantially influenced LLM scoring, generating opposite biases relative to expert evaluation. The novel contribution of this study is that prompt strategy can alter not only the magnitude but also the direction of scoring bias. Calibration approaches, such as context engineering, may help align AI scoring with expert judgment. While AI-generated feedback shows promise for formative assessment, reliable summative use requires careful validation.