EvalAgent : Discovering Implicit Evaluation Criteria from the Web

Manya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, Greg Durrett · 2025

Read EvalAgent : Discovering Implicit Evaluation Criteria from the Web by Manya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, Greg Durrett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15219.txt.

Description

Manya Wadhwa and co-authors present EvalAgent in a 2025 paper, a framework that finds hidden criteria for judging language model writing. Judges usually get a short list of stated criteria, such as accuracy and coherence. Yet a good answer needs more than the basics: a talk draft, for instance, also needs a strong opening and a clear takeaway. Those extra criteria are rarely stated in the user's prompt.

EvalAgent mines expert guidance posted online, then uses that evidence to propose detailed, task specific criteria tied to solid sources. Tests show the criteria it finds are often implicit in the prompt, yet worded with care. First drafts often fail them, but the criteria are actionable, so drafts can be revised to meet them. Mixing EvalAgent criteria with model written ones surfaced more of what human judges value than model criteria alone.

Catalog record

CREATORManya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, Greg Durrett
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE85 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15219

Browse the full library · About the Tower of Babel