AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence
Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao · 2025
Read AXIOM: Benchmarking LLM-as-a-Judge for Code via Rule-Based Perturbation and Multisource Quality Calibration CCS: Software and its engineering CCS: Computing methodologies Artificial intelligence by Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-20159.txt.
Description
Ruiqi Wang and five co-authors benchmark the judges that grade machine-written code. Language models now score code for flaws like hallucinated APIs. But the benchmarks that validate those judges have flaws of their own. Coarse pass-fail labels cannot split deep flaws from minor ones. Hand-scored rubrics are vague and subjective. Fresh test code skews the quality spread. Their frame, AXIOM, builds a balanced benchmark instead.
Programs are scored by the fix-up effort they need to be deployable. Set perturbation rules dial each program to a target quality. A hybrid calibration step then picks diverse programs for lean human scoring, with rich context at hand. The result, AXIOMBENCH, holds nearly two thousand multilingual code problems. Its score spread is far more balanced than past sets.
Catalog record
| CREATOR | Ruiqi Wang, Xinchen Wang, Cuiyun Gao, Chun Yong Chong, Xin Xia, Qing Liao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-20159