Abstract

The emergence of Large Language Models (LLMs) has renewed interest in automated essay scoring, yet a barrier to adoption persists: LLM-generated scores are sensitive to prompt formulation, and it is unclear which aspects of a prompt drive this effect or whether findings from one model or institution transfer to another. For educators and institutions considering deployment, this uncertainty is consequential, as without understanding the structure of prompt sensitivity, there is no principled basis for deciding whether to trust an LLM-generated score. This study evaluates three frontier LLMs (GPT-5.4, Claude Opus 4.6, Gemini 3 Flash) across a 3×3×3 factorial design varying criteria specificity, calibration intervention, and scoring strategy on a calibration sample of 153 undergraduate essays from three UK universities with distinct marking conventions. A main-effects ANOVA on the 27 per-model conditions reveals that prompt sensitivity is substantial but strikingly model-specific: criteria specificity accounts for the largest share of condition-level variance for Gemini (57\%, p < .001), scoring strategy for GPT (35\%, p < .01), and scoring strategy and calibration together for Claude (32\% and 24\% respectively), with no single dimension dominating across models. The optimal configuration also differs across institutions, even within the same model. At each institution, the gap between best and worst configurations spans 4–5 RMSE points on the calibration set, with the worst performing worse than a naïve baseline. For the models tested, these findings indicate that no universal prompt recipe exists, and that responsible deployment requires empirical prompt calibration as a prerequisite rather than an optional refinement - a process more similar to hyperparameter tuning than to simple instruction design.