Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys DOI: 10.1145/3788646.3789533 Conference: Make sure to enter the correct conference title from your rights confirmation email; March 3–5, 2026; Berkeley, CA, USA CS&law ’26: ACM Symposium on Computer Science and Law, March 03–05, 2026, Berkeley, CA, USA ISBN: 979-8-4007-2447-3/26/03 Conference: Symposium on Computer Science and Law; March 3–5, 2026; Berkeley, CA, USA Symposium on Computer Science and Law (CSLAW ’26), March 3–5, 2026, Berkeley, CA, USA DOI: 10.1145/3788646.3789533 ISBN: 979-8-4007-2447-3/26/03 CCS: Applied computing Law CCS: Computing methodologies Natural language processing CCS: Computing methodologies Information extraction
Mohamed Afane, Emaan Hariri, Derek Ouyang, Daniel E. Ho · 2026
Read Benchmarking Legal RAG: The Promise and Limits of AI Statutory Surveys DOI: 10.1145/3788646.3789533 Conference: Make sure to enter the correct conference title from your rights confirmation email; March 3–5, 2026; Berkeley, CA, USA CS&law ’26: ACM Symposium on Computer Science and Law, March 03–05, 2026, Berkeley, CA, USA ISBN: 979-8-4007-2447-3/26/03 Conference: Symposium on Computer Science and Law; March 3–5, 2026; Berkeley, CA, USA Symposium on Computer Science and Law (CSLAW ’26), March 3–5, 2026, Berkeley, CA, USA DOI: 10.1145/3788646.3789533 ISBN: 979-8-4007-2447-3/26/03 CCS: Applied computing Law CCS: Computing methodologies Natural language processing CCS: Computing methodologies Information extraction by Mohamed Afane, Emaan Hariri, Derek Ouyang, Daniel E. Ho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03300.txt.
Description
Mohamed Afane and three colleagues benchmark artificial intelligence tools that survey state laws for lawyers. Their 2026 paper uses LaborBench, built from a slow manual count of every state's unemployment insurance rules by Department of Labor attorneys. Earlier tests found standard retrieval-augmented generation right only about seven times in ten on yes-or-no questions.
Three newer tools face the same test here. A custom research assistant, STARA, does far better, while the Westlaw and LexisNexis products score worse than the older standard method. Close error analysis then surprises everyone: many supposed machine errors were omissions by the human attorneys, lifting STARA's true score further. The paper closes with concrete design principles for accurate legal research across many jurisdictions at once.
Catalog record
| CREATOR | Mohamed Afane, Emaan Hariri, Derek Ouyang, Daniel E. Ho |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94430 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03300