RubricRAG : Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation

Kaustubh D. Dhole, Eugene Agichtein · 2026

Read RubricRAG : Towards Interpretable and Reliable LLM Evaluation via Domain Knowledge Retrieval for Rubric Generation by Kaustubh D. Dhole, Eugene Agichtein (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20882.txt.

Description

Dhole and Agichtein study how language models can write clear scoring rubrics for other model answers. Simple numeric scores do not show what an answer did right or wrong. Hand written rubrics are clearer but too slow to make at scale. They find that off the shelf models write rubrics that align poorly with human ones. Their RubricRAG method pulls domain knowledge from rubrics of related questions at run time. It yields rubrics closer to human ones. It also improves how well good responses are picked out downstream.

Catalog record

CREATORKaustubh D. Dhole, Eugene Agichtein
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37932
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.20882

Browse the full library · About the Tower of Babel