Supporting Humans in Evaluating AI Summaries of Legal Depositions Conference: 2026 ACM SIGIR Conference on Human Information Interaction and Retrieval; March 22–26, 2026; Seattle, WA, USA 2026 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’26), March 22–26, 2026, Seattle
Naghmeh Farzi, Laura Dietz, Dave D. Lewis · 2026
Read Supporting Humans in Evaluating AI Summaries of Legal Depositions Conference: 2026 ACM SIGIR Conference on Human Information Interaction and Retrieval; March 22–26, 2026; Seattle, WA, USA 2026 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’26), March 22–26, 2026, Seattle by Naghmeh Farzi, Laura Dietz, Dave D. Lewis (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-15182.txt.
Introduction
Naghmeh Farzi, Laura Dietz, and Dave D. Lewis help lawyers check AI summaries of legal depositions. Their 2026 arXiv paper deals with a risky new habit. Large language models now sum up long legal documents, but in law a wrong fact can do real harm.
A deposition summary must stay true to the sworn record. The authors build support for humans who evaluate these summaries. Their route rests on nuggets, small key facts pulled from the text. Nugget-based checks have proven helpful for scoring summaries, and the work adapts them to legal needs.
Catalog record
| CREATOR | Naghmeh Farzi, Laura Dietz, Dave D. Lewis |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 19 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-15182
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supporting Humans in Evaluating AI Summaries of Legal Depositions Conference: 2026 ACM SIGIR Conference on Human Information Interaction and Retrieval; March 22–26, 2026; Seattle, WA, USA 2026 ACM SIGIR Conference on Human Information Interaction and Retrieval (CHIIR ’26), March 22–26, 2026, Seattle, WA, USA DOI: 10.1145/3786304.3787923 ISBN: 979-8-4007-2414-5/2026/03 CCS: Information systems Information retrieval Naghmeh Farzi, Laura Dietz, Dave D. Lewis Abstract . While large language models (LLMs) are increasingly used to summarize long documents, this trend poses significant challenges in the legal domain, where the factual accuracy of deposition summaries is crucial. Nugget - based methods have been shown to be extremely helpful for the automated evaluation of summarization approaches. In this work, we translate these methods to the user side and explore how nuggets could directly assist end users. Although prior systems have demonstrated the promise of nugget - based evaluation, its potential to support end users remains underexplored. Focusing on the legal domain, we present a prototype that leverages a factual nugget - based approach to support legal professionals in two concrete scenarios: (1) determining which of two summaries is better, and (2) manually improving an automatically generated summary. 1 1 1 Prototype available at https://github.com/TREMA-UNH/human-ai-legal-summaries † † cc-license: by 1. Introduction Deposition transcripts are verbatim records of witness testimony taken under oath during pre - trial discovery. A single transcript often exceeds hundreds of pages and contains dense question and answer exchanges. Legal professionals must manually extract, synthesize, and validate the critical facts, a process that is time - consuming, error - prone, and cognitively demanding. Conflicting testimony, inconsistent statements, and the need to cross - reference other depositions or exhibits compound the difficulty. Cognitive fatigue and tight deadlines increase the risk that essential details are missed, creating a major efficiency bottleneck. Large language models (LLMs) have opened new possibilities for automated text analysis, but may contain hallucinations ( 19 ) , factual inaccuracies, and a lack of transparency, which necessitates rigorous verification ( 20 ) . Explainable and transparent AI is therefore essential in the legal domain. We focus on early-stage litigation review, where AI-generated deposition summaries support case assessment. In this setting, legal professionals face two high-cognitive-load scenarios when using AI-generated summaries: (1) Determining which of two AI-generated summaries is better, often arising from different summary styles (e.g., chronological vs. narrative), prompting strategies, model versions, or LLM non-determinism; (2) Manually improving an AI-generated summary. Currently, both tasks are performed without system support. The second requires exhaustive cross - checking against the transcript, and the first involves direct comparison of the summaries together with verification against the source. Recent advances in evaluating summarization and retrieval-augmented generation (RAG) systems focus on system - level performance but provide a promising foundation for user - focused assistance. Traditional metrics such as ROUGE ( 5 ) and BLEU ( 11 ) rely on lexical overlap, while manually aligned nuggets have been used for finer - grained evaluation ( 6 ) . Recent automatic evaluation methods use nuggets with LLM-based alignment to evaluate the recall of facts ( 2 ; 17 ; 13 ; 14 ) . Many of these newer methods employ an “LLM-as-a-Judge” framework, which uses LLMs as automated evaluators to assess AI responses against human-like criteria ( 21 ; 4 ) . Although effective for model evaluation, these approaches have not yet been adapted to directly support end - user tasks. In this work, we leverage nugget - based evaluation to help legal professionals verify, compare, and refine AI - generated summaries. Contributions. We present a demonstration prototype that integrates automatic nugget extraction and alignment, and citation verification, are integrated into a user interface that supports legal experts in two cognitively demanding scenarios: (1) determining which of two AI summaries is better, and (2) manually improving an AI - generated draft. 2. Related Work Our work sits at the intersection of legal information retrieval (IR) and extraction, and human-in-the-loop systems. We ground our contributions in a review of existing literature, identifying persistent challenges that our framework is designed to address. 2.1. Legal Text Summarization Early efforts in legal summarization primarily focus on extractive methods, where key sentences are selected from a source document. Systems like CaseSummarizer ( 12 ) exemplify this approach, which uses word frequency and domain-specific rules. The advent of deep learning led to abstractive summarization models, with Norkute et al. ( 10 ) using a Pointer Generator network and Zhong et al. ( 22 ) exploring an approach named ’train-attribute-mask’ to identify predictive sentences. However, as they noted, focusing solely on predictive sentences may not capture all legally relevant facts, while summarizing legal meetings, depositions, and court transcripts has been largely overlooked, underscoring a gap in domain-specific applications ( 1 ) . 2.2. Hallucination and Factual Inaccuracy The move toward fact-based evaluation is driven by the risk of hallucination and factual inaccuracy, which is a major concern in high-stakes domains such as legal. A landmark study by Magesh et al. ( 8 ) revealed that leading AI legal research tools, such as Casetext, Thomson Reuters, or LexisNexis, despite using advanced Retrieval-Augmented Generation (RAG) techniques, produced hallucinated content in over 17% of the time. Similarly, Savelka et al. ( 15 ) confirmed that direct use of models like GPT-4, without retrieval-augmentation, resulted in factual inaccuracies. These findings highlight that unsupervised AI systems are not yet suitable for critical legal tasks without verification, a sentiment supported by Akter et al. ( 1 ) who noted the risk of missing or misrepresenting critical facts in legal summarization. Accordingly, we treat hallucinations as inherent and focus on surfacing omissions and unsupported claims to aid human verification. 2.3. Evaluating Legal Summaries A critical finding across the literature is that standard automated metrics such as ROUGE ( 5 ) and BLEU ( 11 ) are inadequate for evaluating legal summaries. These metrics offer little explainability beyond surface-level word matching and fail to capture paraphrases, synonyms, and some other critical dimensions of quality, such as contextual fidelity ( 16 ) , factual accuracy, and traceability to the source text. This has spurred a move toward fact-based evaluation frameworks. Some of the recent work relies on using questions and answers to do the evaluation task. Farzi and Dietz ( 3 ) use questions as a rubric and their answerability against the document as a way to evaluate the LLM or retriever’s output. Xu and Ashley ( 18 ) proposed a question-answering approach to evaluate legal case summaries that uses an LLM to generate and grade question and answer pairs from summaries, relying on comparing answers derived from the human-written reference summary with answers generated from the model-generated summary to assess quality. ARGUE ( 9 ) and its automated implementation Auto-ARGUE ( 17 ) , while not specifically for legal domain evaluation, focus on metrics such as sentence precision and nugget recall, by an LLM prompted to make binary judgments on whether a given fact is present or a claim is supported by its citation. While prior work uses fact-based rubrics to generate static evaluation scores (e.g., nugget recall), our system uses its nugget-based rubric to enable two user-centered workflows to support them for summary comparison and refinement. 2.4. Human-in-the-Loop (HITL) for Verification The