EvalAgent : Discovering Implicit Evaluation Criteria from the Web
Manya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, Greg Durrett · 2025
Read EvalAgent : Discovering Implicit Evaluation Criteria from the Web by Manya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, Greg Durrett (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15219.txt.
Description
Manya Wadhwa and co-authors present EvalAgent in a 2025 paper, a framework that finds hidden criteria for judging language model writing. Judges usually get a short list of stated criteria, such as accuracy and coherence. Yet a good answer needs more than the basics: a talk draft, for instance, also needs a strong opening and a clear takeaway. Those extra criteria are rarely stated in the user's prompt.
EvalAgent mines expert guidance posted online, then uses that evidence to propose detailed, task specific criteria tied to solid sources. Tests show the criteria it finds are often implicit in the prompt, yet worded with care. First drafts often fail them, but the criteria are actionable, so drafts can be revised to meet them. Mixing EvalAgent criteria with model written ones surfaced more of what human judges value than model criteria alone.
Catalog record
| CREATOR | Manya Wadhwa, Zayne Sprague, Chaitanya Malaviya, Philippe Laban, Junyi Jessy Li, Greg Durrett |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 85 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-15219