A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction CCS: Computing methodologies Natural language generation CCS: Computing methodologies Information extraction CCS: Applied computing Law CCS: Applied computing Document searching
Nicole Lincoln, Nick Whitehouse, Jaron Mar, Rivindu Perera · 2026
Read A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction CCS: Computing methodologies Natural language generation CCS: Computing methodologies Information extraction CCS: Applied computing Law CCS: Applied computing Document searching by Nicole Lincoln, Nick Whitehouse, Jaron Mar, Rivindu Perera (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-05532.txt.
Introduction
Small, specialized models can beat giant ones at narrow jobs, argue Nicole Lincoln, Nick Whitehouse, Jaron Mar, and Rivindu Perera. Their test case is pulling structured fields out of legal contracts. They pitted Olava Extract, a self-hosted legal model, against five frontier systems on contracts annotated by legal professionals.
Olava Extract posted the best overall scores in the study, with the highest precision as well. Fewer invented or unsupported fields matter in legal work, where errors create review burden and risk. Its running costs were also far lower, falling by roughly four-fifths to nearly all of the frontier models' cost.
Catalog record
| CREATOR | Nicole Lincoln, Nick Whitehouse, Jaron Mar, Rivindu Perera |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.05532
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Few Good Clauses: Comparing LLMs vs Domain-Trained Small Language Models on Structured Contract Extraction CCS: Computing methodologies Natural language generation CCS: Computing methodologies Information extraction CCS: Applied computing Law CCS: Applied computing Document searching Nicole Lincoln, Nick Whitehouse, Jaron Mar, Rivindu Perera Abstract . This paper evaluates whether a domain-trained Small Language Model (SLM) can outperform frontier Large Language Models on structured contract extraction while operating at dramatically lower cost. We test Olava Extract, a self-hosted legal-domain Mixture-of-Experts model, against five commercially available frontier systems on 24 held-out public contracts annotated by legal professionals, covering 508 labelled field instances across 26 extraction targets. Olava Extract achieved the strongest aggregate performance in the study, with a macro F1 of 0.812 and a micro F1 of 0.842, outperforming all evaluated frontier baselines while operating at a fraction of their inference cost. Under batched inference, extraction cost fell by 78% to 97% compared with the frontier models tested. Olava Extract also achieved the highest precision scores in the evaluation, producing fewer unsupported extractions and fewer hallucinated answers than the frontier baselines. This distinction is particularly important in legal workflows, where hallucinated outputs can create significant downstream review burden, operational risk, and loss of trust in automated systems. The implications extend beyond contract extraction itself. For legal services, the results suggest that high-performing, human-comparable legal AI can now operate at infrastructure-scale throughput and materially lower cost using smaller domain-trained systems, increasing the economic pressure on traditional review models built around Junior Lawyers and Legal Process Outsourcers. More broadly, the findings challenge a central assumption of the current AI market: that commercially valuable enterprise AI capability must remain tightly coupled to ever-larger models, massive infrastructure expenditure, and centrally hosted providers. On a commercially meaningful enterprise task, a relatively compact specialised model was able to compete directly with frontier-scale systems while dramatically outperforming them on cost and deployment efficiency. 1. Introduction Extracting key terms from contracts is one of the most time-consuming tasks in legal operations. Agreements vary widely in structure, terminology, drafting style. Relevant information may appear in clauses, schedules, exhibits, or definitions spread across many pages. Reviewers must identify provisions, dates, parties, obligations, and financial terms while preserving enough source context for the extracted answer to be verified. Recent work has shown that Large Language Models (LLMs) can automate parts of this work. In Better Call GPT ( Martin et al. 2024 ) , we evaluated frontier LLMs against Junior Lawyers and Legal Process Outsourcers (LPOs) on contract review. The models matched or exceeded human accuracy while reducing review time and cost. In Better Bill GPT ( Whitehouse et al. 2025 ) , we extended the comparison to legal invoice review and found that LLMs outperformed human reviewers across the evaluated decision and classification tasks. Those results show that frontier LLMs can perform structured legal analysis tasks well. They also leave open an important deployment question. The models evaluated in those studies were large, commercially hosted systems accessed through third-party APIs. For legal teams, that setup can create practical constraints around data control, security review, cost predictability, model versioning, and long-term availability. Small Language Models (SLMs) offer a new path. They can be fine-tuned on domain-specific data and deployed within an organisation’s own infrastructure. If such a model can meet frontier-model quality on contract extraction, it may offer a better fit for workflows where confidentiality, cost control, and deployment stability matter. In this paper, we use "SLM" to include efficient Mixture-of-Experts (MoE) models whose active inference footprint is in the range commonly associated with smaller dense models, even when their nominal parameter count is higher. This study evaluates Olava Extract, a domain-trained contract extraction model. We test it on a full-document structured extraction task covering 26 common contract fields across 24 held-out public contracts, with 508 human-labelled field instances. The fields include extracted clauses, duration fields, dates, currency values, Boolean and categorical selections, and short text fields identifying information that is either directly stated or inferred from the contract. We compare Olava Extract with five frontier models using the same contract set, output schema, prompt structure, inference approach, and scoring rules. The study focuses on three questions: (1) Can a domain-trained Small Language Model reach aggregate extraction performance comparable to leading frontier models? (2) Are Small Language Models cheaper than Large Language Models, beyond the obvious advantages of self-hosting? (3) Are Small Language Models faster than Large Language Models? Together, these questions extend the research direction established in Better Call GPT and Better Bill GPT . In Better Call GPT , we posited that LLMs would disrupt Junior Lawyers and LPOs by showing that they could perform legal review tasks with comparable accuracy, greater speed, and lower cost. This study continues that experimentation by examining a high-volume task: structured contract extraction. The focus now shifts from model capability to production viability. Frontier LLMs can perform legal tasks with strong accuracy, but at the scale required for structured contract extraction, their cost, latency, and data-control constraints may reduce their practical advantage over human review. This paper evaluates whether a smaller, domain-trained model can retain sufficient extraction accuracy while providing a more cost-effective and operationally practical path to scaled legal AI deployment. 2. Related Work This study builds on three strands of prior work: legal-domain model adaptation, contract-specific extraction benchmarks, and efficient fine-tuning of smaller models. Prior legal NLP research has shown that legal text benefits from domain-specific modelling. Chalkidis et al. 2020 evaluated strategies for adapting BERT to legal corpora and released a family of legal-domain BERT models for legal NLP applications ( Chalkidis et al. 2020 ) . This supports the core premise that legal language is specialised enough to justify domain adaptation rather than relying only on general-purpose models. Contract review has also become a useful benchmark for legal AI because it requires both structured information extraction with long, variable documents. CUAD introduced an expert-annotated contract-review dataset, framing review as the identification of contract provisions that matter to human reviewers ( Hendrycks et al. 2021 ) . ContractNLI approached contract analysis as document-level natural language inference, requiring models to classify hypotheses as entailed, contradicted, or not mentioned and to identify supporting evidence spans ( Koreeda and Manning 2021 ) . Together, these datasets show that contract analysis is automatable but remains difficult, especially where documents are long, drafting conventions vary, or evidence must be gathered across multiple clauses. Recent legal-industry evaluations have shown that frontier LLMs can perform structured legal review tasks at or above human-reviewer baselines. Prior studies in this series found strong LLM performance on procurement-contract issue spotting and legal-invoice review, with large gains in speed and cost efficiency ( Martin et al. 2024 ; Whitehouse et al. 2025 ) . Those studies establish the usefulness of LLMs for structured legal workflows, but they rely on large, API-hosted frontier models. The present study focuses on whether similar utility can be achieved with a smaller, self-hostable model. Recent work outside law shows that compact models trained on carefully curated data can rival much larger systems on standard benchmarks, as demonstrated by Phi-3 ( Abdin et al. 2024 ) . In the legal domain, SaulLM-7B shows that continued training on legal corpora