Gavel : Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization
Yao Dou, Wei Xu · 2026
Read Gavel : Agent Meets Checklist for Evaluating LLMs on Long-Context Legal Summarization by Yao Dou, Wei Xu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-04424.txt.
Description
Yao Dou, Benjamin Mamut, and Wei Xu build a sterner examiner for language models that summarize legal cases, in a 2026 arXiv paper. Modern models swallow a million tokens, but a single case can sprawl past a hundred thousand across many documents, and nobody knew where the summaries fail. Their Gavel framework grades with checklists, leftover-fact checks, and style review, while a companion agent grades factual coverage straight from the source documents, with no reference summary needed.
Across twelve frontier models, the main sin is omission, not invention: filings and dates come easily, settlements and rare complications get dropped, and quality fades as cases lengthen. Validating the grader took one hundred sixty hours of human annotation. The agent version matches other methods while using at least a third fewer tokens, and it transfers to medical documents with even larger savings.
Catalog record
| CREATOR | Yao Dou, Wei Xu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-04424