Amulet : Putting Complex Multi-Turn Conversations on the Stand with LLM Juries

Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, Xiang Ren · 2025

Read Amulet : Putting Complex Multi-Turn Conversations on the Stand with LLM Juries by Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, Xiang Ren (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20451.txt.

Description

Sahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, and Xiang Ren present Amulet, a framework that improves how language models judge other models' conversations. Their 2025 paper deals with complex, multi-turn chats.

Models are widely used as judges of model output, so their accuracy matters. Real conversations are long and shift across topics, intents, and requirements. Amulet draws on linguistic ideas: dialog acts, which capture communicative structure and intent, and maxims, the principles of good conversation. It feeds both kinds of insight into the judgment. On four hard datasets, the framework improved judge accuracy on preference data with long conversational context.

Catalog record

CREATORSahana Ramnath, Anurag Mudgil, Brihi Joshi, Skyler Hallinan, Xiang Ren
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE105 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-20451

Browse the full library · About the Tower of Babel