Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law

Parisa Kordjamshidi, Samer Aslan, Madhavan Seshadri, Leslie Barrett, Enrico Santus · 2026

Read Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law by Parisa Kordjamshidi, Samer Aslan, Madhavan Seshadri, Leslie Barrett, Enrico Santus (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16052.txt.

Introduction

Kordjamshidi and four colleagues test whether language models truly reason about tax law or merely recall tainted training data. They build a detection protocol and a fresh test suite of unseen cases and rule changes. Stand-alone models look inflated by taint. Hybrid neuro-symbolic systems, which turn statutes into formal rules for a symbolic solver, generalize better.

Catalog record

CREATORParisa Kordjamshidi, Samer Aslan, Madhavan Seshadri, Leslie Barrett, Enrico Santus
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62773
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.16052

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reasoners or Translators? Contamination-aware Evaluation and Neuro-Symbolic Robustness in Tax Law Parisa Kordjamshidi, Samer Aslan, Madhavan Seshadri, Leslie Barrett, Enrico Santus Abstract Recent advances in large language models (LLMs) have significantly enhanced automated legal reasoning. Yet, it remains unclear whether their performance reflects genuine legal reasoning ability or artifacts of data contamination. We present a comprehensive empirical study of tax law reasoning approaches and implement a contamination detection protocol to rigorously assess LLM reliability. We show that performance can be inflated by contamination. Building on this analysis, we conduct a systematic evaluation, comparing monolithic LLMs with hybrid systems that translate statutory text into formal representations and delegate inference to symbolic solvers. We build a novel test suite designed to probe generalization to unseen documents via case and rule variations. Our findings indicate that legal reasoning is inherently compositional and that neuro-symbolic frameworks offer a more reliable and robust foundation for legal AI, as well as improved generalization to unobserved situations. 1 Introduction Legal reasoning is a complex and sensitive task that requires rigorous, explainable, and verifiable frameworks, as errors can have severe and irreversible consequences. For example, mistakes in tax returns may affect credit, student aid eligibility, and health coverage. While recent large language models (LLMs) have shown impressive capabilities ( 2 ; 22 ) , their performance on legal reasoning tasks remains inconsistent ( 4 ; 11 ) and prone to hallucination ( 8 ; 18 ) . Reported accuracies vary widely across benchmarks, with some nearing 90% ( 20 ) and others falling below practical thresholds ( 21 ) . More broadly, inflated performance in LLM evaluations is often attributed to data contamination and memorization ( 10 ) , raising concerns about benchmark validity and undermin1ing reliability in genuinely novel legal scenarios, where generalization to unseen statutes, fact patterns, and interpretations is essential. Figure 1: Neuro-symbolic pipeline for tax-law reasoning. (A) Case formalization: an LLM converts a textual fact pattern into a set of Prolog facts (structured case knowledge). (B) Logical reasoning: Given the Prolog representation of the statute excerpt and a query, the Prolog solver executes the composed program (case facts + statute rules + query) to produce either a Boolean decision (entailment vs. contradiction) or a numeric answer (e.g., tax owed). Recent work in LLM-based legal reasoning has focused on making reasoning explicit through neuro-symbolic models to improve verification and interpretability. These approaches combine symbolic reasoning, which is explainable and verifiable, with neural models capable of handling complex, real-world data. In tax law, this is typically achieved by coupling LLMs with symbolic engines such as Prolog, SAT solvers ( 13 ; 9 ) , and domain-specific languages ( 17 ) . More broadly, the integration of general solvers and theorem provers has been proposed to further enhance formal reasoning capabilities of LLMs ( 15 ; 23 ) . A central bottleneck, however, is translating natural language into formal logic. While such formalization has long been viewed as a principled foundation for verifiability and accountability in legal analytics ( 1 ) , earlier studies have shown that scaling logical reasoning to real-world applications is challenging ( 26 ) . More recent work in the LLM era has further highlighted the difficulty of mapping natural language to formal logic ( 31 ; 24 ) . Although advances in semantic parsing with LLMs have improved legal text annotation ( 27 ) , empirical results in tax law show that full formal translation remains challenging. In some case studies, pipelines relying on explicit formalization can be outperformed by direct natural language reasoning ( 16 ) , highlighting limitations of current approaches in realistic settings. Despite these limitations, a key advantage of this line of work is enabling verification and effective human-in-the-loop intervention. Prior work shows that validating generated logical programs and deferring uncertain cases to experts can substantially reduce tax penalties ( 16 ) , underscoring the value of explicit reasoning even when full automation is not feasible. Agentic frameworks further support feedback, verification, and self-correction, improving the robustness of hybrid neuro-symbolic systems ( 12 ) . In this paper, we present a systematic study of legal reasoning in tax law. We revisit existing approaches and reproduce prior results using state-of-the-art LLMs. Motivated by the risks of data contamination, we compare the reasoning abilities of monolithic LLMs with neuro-symbolic models and revisit these approaches. Specifically, we adopt a modular approach that involves translating textual cases into logical facts, maintaining a knowledge base (KB) of logical rules, and forming logical programs by combining these facts and rules. Figure 1 illustrates the query ”How much tax should Alice pay in 2015?” , translated to owes_{t}ax(alice,2015,_{)} in Prolog. Executing this query over the formalized statutes and case resolves the unknown argument, yielding a tax value of 14{,}000 . In this setup, LLMs serve as translation tools and deductive reasoning is handled by Prolog. The goal of our extensive empirical evaluation is to address: (Q1) the effectiveness of monolithic vs. neuro-symbolic approaches, (Q2) alignment with prior results and the performance of newer state-of-the-art LLMs, (Q3) the impact of data contamination, and (Q4) generalization under case and rule perturbations. Our findings show that LLM performance is partly inflated by contamination and lacks stability under variations. In contrast, neuro-symbolic models are more robust and reliable, achieving better performance under rule and data changes. We conclude that legal reasoning remains a compositional and complex task and advocate for hybrid neuro-symbolic systems that combine LLM capabilities in parsing and formal translation with symbolic reasoning engines to improve generalization, interpretability, and verifiability. 2 Related Work Legal reasoning has become an active area of research with the rise of LLMs. It provides a rich test bed for diverse reasoning types, including case-based (analogical), abductive, and deductive reasoning ( 21 ) , often used in combination. Consequently, numerous benchmarks, datasets, and model architectures have been proposed to evaluate legal reasoning systems ( 5 ; 11 ; 30 ) . Here, we focus on prior work in tax law. We choose this domain due to the availability of human annotations and its structured, rule-based nature, making it well suited for neuro-symbolic modeling. In tax law, the most widely used dataset is SARA ( 13 ) , which provides human-annotated logical forms of rules and cases. Recent LLM performance has been evaluated by 14 across legal tasks in Chinese and English, including tax law reasoning on SARA , highlighting challenges such as hallucinations and limited legal knowledge. The use of symbolic solvers in law dates back to early AI research ( 3 ) , with logic programming and rule-based systems widely applied to legal reasoning ( 29 ; 28 ) . Much of this work emerged from argumentation mining and computational argumentation research ( 19 ; 6 ) . More recent research focuses on translating legal text into formal logic, using LLMs to obtain executable representations. In this direction, 16 propose a Prolog-based framework, arguing that verifiable programs with human-in-the-loop supervision can mitigate monetary losses from tax computation errors. With the rise of agentic AI, integrating formal logic into agentic frameworks has been explored ( 25 ) . In this approach, cases are translated into formal logic, and an SMT solver computes numerical

Read the complete text