Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification
2026 25th International Conference on Machine Learning and Applications (ICMLA), 2026
Abstract
Large language models (LLMs) are increasingly integrated into complex workflows as automated evaluators, yet their reliability in assessing unconventional reasoning processes remains under-explored. Current evaluation frameworks often overlook models' robustness to procedural equivalence, the ability to recognize valid but non-canonical paths to a correct result. In this work, we introduce a benchmark specifically designed to test LLMs in an evaluative capacity, using linear-equation problems with diverse solution variants as a controlled proxy for correctness-critical, multi-step verification tasks in science and engineering. We further define a process-centric evaluation framework across three dimensions: (i) final-answer correctness, (ii) step-level correctness, and (iii) localization of the initial logical error. Our evaluation of state-of-the-art open LLMs reveals a significant robustness gap: models that accurately evaluate canonical solutions often fail when presented with perturbed but logically equivalent variants. Models exhibit high false-negative rates by rejecting valid alternative solutions and show signs of bias. Our results suggest that while adaptation strategies can narrow this gap, achieving reliable process-level verification remains a critical challenge for deploying LLM evaluators in correctness-critical domains.
Cite this work
Fateme Mazdarani and Carlos Toxtli-Hernández. 2026. Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification. 2026 25th International Conference on Machine Learning and Applications (ICMLA).
@inproceedings{Mazdarani2026Beyond,
title = {Beyond the Answer Key: Robustness Evaluation of Large Language Models for Step-Level Mathematical Verification},
author = {Mazdarani, Fateme and Toxtli, Carlos},
booktitle = {2026 25th International Conference on Machine Learning and Applications (ICMLA)},
publisher = {IEEE},
year = {2026},
note = {Forthcoming},
url = {https://www.icmla-conference.org/icmla26/}
}Related
