Automated Validation of LLM-based Evaluators for Software Engineering Artifacts
Ora Fandina, Eitan Farchi, Shmulik Froimovich · 2025
Read Automated Validation of LLM-based Evaluators for Software Engineering Artifacts by Ora Fandina, Eitan Farchi, Shmulik Froimovich (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02827.txt.
Description
Ora Nova Fandina and co-authors introduce REFINE, a frame to test language-model evaluators for software work without human judges, in a 2025 paper. Models now write, review, and grade code, but proving that a model grader is reliable is hard. Human grading is costly, subjective, and cannot scale. Past automated checks miss fine gaps in quality between code artifacts.
REFINE has two parts. One part builds data sets of artifacts whose quality drops step by step in a controlled way. The other part scores each candidate grader by how closely its ranking matches the known order. Users can tune how fine the quality steps are. That control lets teams stress-test graders at the level of detail their work needs.
Catalog record
| CREATOR | Ora Fandina, Eitan Farchi, Shmulik Froimovich |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-02827