Deterministic Fuzzy Triage for Legal Compliance Classification and Evidence Retrieval

Rian Atri · 2026

Read Deterministic Fuzzy Triage for Legal Compliance Classification and Evidence Retrieval by Rian Atri (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07390.txt.

Introduction

Atri offers legal teams a deterministic, transparent alternative to opaque models for triaging contractual evidence. Thresholds are tuned to maximize automatic coverage under a tight error cap on auto-decided cases. A dual-encoder model trained on graded clause retrieval and compliance labels maps each score into three bands: auto-noncompliant, auto-compliant and human-review. The result is a seed-stable system described by a few scalar parameters, bridging hand rules and large black-box models.

Catalog record

CREATORRian Atri
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41538
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07390

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Deterministic Fuzzy Triage for Legal Compliance Classification and Evidence Retrieval Rian Atri Abstract Legal teams now use ML to triage large volumes of contractual evidence, but most models are opaque, non-deterministic, and hard to align with frameworks such as HIPAA or NERC–CIP. We study a simple, reproducible alternative based on deterministic dual encoders and transparent fuzzy triage bands. Concretely, we train a RoBERTa-base dual encoder with a 512-dimensional projection and cosine similarity on the ACORD benchmark for graded clause retrieval, and then fine-tune it on a CUAD-derived binary compliance dataset. Across five random seeds (40–44) on a single NVIDIA A100 GPU, our model achieves ACORD-style retrieval performance of approximately NDCG@5 ≈ 0.38 \approx 0.38 – 0.42 0.42 , NDCG@10 ≈ 0.45 \approx 0.45 – 0.50 0.50 , and 4-star Precision@5 ≈ 0.37 \approx 0.37 on the test split. On CUAD-derived binary labels, we obtain AUC ≈ 0.98 \approx 0.98 – 0.99 0.99 and F 1 ≈ 0.22 {}_{1}\approx 0.22 – 0.30 0.30 depending on the positive-class weight, substantially outperforming majority and random baselines on a highly imbalanced setting (positive rate ≈ 0.6 % \approx 0.6\% ). On top of the scalar compliance scores, we introduce a simple fuzzy triage mapping that partitions the score axis into three regions: auto-noncompliant, auto-compliant, and human-review. We tune the lower and upper thresholds on validation data to maximize auto-decision coverage subject to a hard constraint on the empirical error rate (at most 2 % 2\% ) over auto-decided examples. This yields deterministic, seed-stable models whose behavior can be summarized by a small number of scalar parameters and reported consistently across runs. We argue that this combination of deterministic encoders, calibrated fuzzy bands, and explicit error constraints offers a practical middle ground between hand-crafted rules and fully opaque large language models: it supports explainable evidence triage, enables reproducible audit trails, and provides a concrete interface for mapping scores and triage regions onto legal concepts such as access control, risk-based review, and residual-risk handling under regulatory frameworks like HIPAA. Introduction Motivation Evidence-Triaged Compliance, not One Shot (OSL) LLMs: Audit, risk, and compliance (GRC) teams are increasingly responsible for justifying decisions that span hundreds to thousands of pages of contracts, policies, and email threads. In many regulated settings (e.g., financial services, insurance, healthcare), audit findings must be traceable back to specific clauses and controls, and reviewers are expected to explain not only what decision was made but also why . Today this work is often carried out through manual search over PDFs, shared drives, and ticketing systems, which does not scale with the number of frameworks, controls, and vendors involved. Recent “AI copilot” interfaces promise to short-circuit this process by letting users “ask a chatbot” to summarize obligations or answer compliance questions over contracts. However, both practitioners and regulators have raised concerns about hallucinations, inconsistent confidence, and the lack of robust uncertainty estimates in such systems. In practice, teams rarely treat these copilots as authoritative decision-makers. Instead, they are used as rough drafting tools, with humans still responsible for reconstructing evidence trails and verifying clause-level compliance. In this work, we focus on a setting that is closer to how compliance teams actually operate: a system that surfaces concrete clause-rule alignments; exposes calibrated scores and triage regions instead of a single yes/no answer; and is explicitly designed to integrate into evidence workflows rather than replace human review. In practice, the evidence needed to support these decisions is scattered across email threads, ticketing systems, contract PDFs, and ad-hoc spreadsheets and folders, and calls for explainable, accountable AI emphasize documenting how such evidence underpins decisions ( Doshi-Velez and Kim 2017 ; Rudin 2019 ; Jobin et al. 2019 ) . Problem Mapping Long-Tail Contract Clauses to Compliance Rules: We study the problem of mapping contract clauses to structured compliance rules. Formally, given a textual rule q (e.g., a control requirement) and a candidate clause c from a contract or policy, the system must (i) estimate how relevant c is to q on a graded scale, and (ii) decide whether c provides sufficient evidence to treat q as compliant or noncompliant. This problem exhibits several challenging properties: • Long-tail clause patterns. The same control can be satisfied by widely varying clause templates across counterparties, jurisdictions, and contract types. • Extreme label imbalance For many rules, only a small fraction of clauses are actually relevant. Even among relevant clauses, only some provide strong, unambiguous evidence of compliance. • High-stakes decisions Mistakes can lead to fines, penalties, and costly remediation efforts in regulated industries ( Jobin et al. 2019 ; Brundage et al. 2020 ; Raji et al. 2020 ) , so practitioners are unwilling to delegate everything to a black-box model. These properties motivate models that are both graded (to capture varying degrees of relevance) and triage-aware (to separate auto-decisions from cases that must be escalated to humans). Limitations of Existing Approaches Prior work on contract understanding and legal NLP has made significant progress on clause tagging and question answering over contracts, for example by framing tasks as span extraction, sequence labeling, or binary classification on datasets such as CUAD. However, these approaches typically learn a single binary decision boundary and are evaluated only on aggregate metrics such as F1 and AUC on a balanced or stratified test set. Likewise, many retrieval-augmented generation (RAG) systems ( Lewis et al. 2020b ) focus on optimizing retrieval metrics for open-domain question answering, but they often treat relevance as binary and rarely expose graded relevance scores or abstention mechanisms at the user interface. In high-stakes compliance settings, two gaps are particularly salient: 1. Lack of graded supervision. Most clause-level datasets do not distinguish between “somewhat relevant” and “highly on-point” evidence, even though practitioners care deeply about this distinction. 2. Lack of explicit triage. Standard binary classifiers and RAG systems do not provide a principled way to say “I am unsure; this should be reviewed by a human,” forcing users to either over-trust or ignore model outputs. Our goal is to address these gaps by combining graded relevance supervision from an insurance benchmark (ACORD) with a fuzzy triage head that explicitly partitions predictions into auto-compliant , auto-noncompliant , and review regions, while remaining compatible with existing clause classification workflows. Contributions Summarizing, our contributions are: (1) a simple, reproducible dual-encoder baseline for legal clause–rule retrieval with graded ACORD supervision; (2) a positivity-weighted classifier and fuzzy triage head that expose explicit coverage–error trade-offs on a CUAD-style binary compliance task; and (3) a discussion of how deterministic, triage-aware models better support explainability, fairness analysis, and regulatory alignment than one-shot LLM copilots, grounded in controls such as HIPAA §164.312 and NERC–CIP. Why not just use an LLM copilot over RAG? One might ask why a relatively small dual encoder is needed when large language models (LLMs) combined with retrieval-augmented generation ( Lewis et al. 2020b ) can already answer compliance questions over contracts. In our setting, however, legal defensibility requires more than aggregate accuracy: regulators and opposing experts must be able to rerun the pipeline offline, obtain identical scores and triage bands, and inspect the small set of scalar parameters that govern behaviour. Proprietary LLM copilots, even when wrapped around a retriever, are typically non-deterministic (due to sampling), updated over

Read the complete text