Abstract
Venues and vendors need a credible evaluator when they compare language-model tools that revise manuscripts from peer reviews, yet a scalar LLM judge is often used without evidence that its preferences track useful revision. We study one such judge-and-rubric pipeline on revision trajectories drawn from multiple machine-learning venue years, comparing its scores with document-level similarity to the authors' actual next versions. The two measures order the machine conditions almost oppositely. A second judge from a different model family and serving stack reproduces the condition ranking, which demonstrates stability of the shared rubric-plus-judge pipeline rather than its validity. More importantly, the method preferred by both judges emits truncated, partial manuscripts in about one in seven runs. The retained logs cannot show whether judges reward those individual failures, and formatting differences confound comparisons with author revisions. Our case study, Prompt-Level Supervisory Alignment, produces documents closer to the next author version than the alternatives, but similarity rewards unchanged text and the decisive editing controls remain absent. The supported recommendation is therefore specific: venues should validate revision evaluators on real trajectories and known completeness failures before using them to certify writing tools.
Cite this work
Carlos Toxtli-Hernández and Manuel Delaflor. 2026. Evaluator Disagreement in AI-Assisted Manuscript Revision. AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI Workshop at NeurIPS 2026.
@inproceedings{Toxtli2026Evaluator,
title = {Evaluator Disagreement in AI-Assisted Manuscript Revision},
author = {Toxtli, Carlos and Delaflor, Manuel},
booktitle = {AI-Native Academia: Authorship, Peer Review, and Conference Governance under AI Workshop at NeurIPS 2026},
address = {Atlanta, GA},
year = {2026},
month = dec,
note = {Poster},
url = {https://openreview.net/forum?id=KcjksLJoBz}
}
