Amulet : Putting Complex Multi-Turn Conversations on the Stand with LLM Juries
Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, Xiang Ren · 2025
Read Amulet : Putting Complex Multi-Turn Conversations on the Stand with LLM Juries by Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, Xiang Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20451.txt.
Description
Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, and Xiang Ren present Amulet, a framework that improves how language models judge other models' conversations. Their 2025 paper deals with complex, multi-turn chats.
Models are widely used as judges of model output, so their accuracy matters. Real conversations are long and shift across topics, intents, and requirements. Amulet draws on linguistic ideas: dialog acts, which capture communicative structure and intent, and maxims, the principles of good conversation. It feeds both kinds of insight into the judgment. On four hard datasets, the framework improved judge accuracy on preference data with long conversational context.
Catalog record
| CREATOR | Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, Xiang Ren |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 105 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20451