Can LLMs Identify Tax Abuse?

Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme · 2025

Read Can LLMs Identify Tax Abuse? by Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-20097.txt.

Introduction

Andrew Blair-Stanek, Nils Holzenberger and Benjamin Van Durme ask whether large language models can spot tax abuse in United States law. They test the best models on real tax plans meant to cut tax for wealthy clients. The task is hard even for expert humans.

Models were asked to check a plan, fill gaps in a partial plan, and build a full plan from scratch. Tax law spans hundreds of thousands of pages of statutes, cases and guidance, and it changes often. One model-based search found a wholly new tax strategy. The authors say such tools could change how tax agencies fight abuse.

Catalog record

CREATORAndrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-20097

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Identify Tax Abuse? Andrew Blair-Stanek, Nils Holzenberger, Benjamin Van Durme Abstract We investigate whether large language models can discover and analyze U.S. tax-minimization strategies. This real-world domain challenges even seasoned human experts, and progress can reduce tax revenue lost from well-advised, wealthy taxpayers. We evaluate the most advanced LLMs on their ability to (1) interpret and verify tax strategies, (2) fill in gaps in partially specified strategies, and (3) generate complete, end-to-end strategies from scratch. This domain should be of particular interest to the LLM reasoning community: unlike synthetic challenge problems or scientific reasoning tasks, U.S. tax law involves navigating hundreds of thousands of pages of statutes, case law, and administrative guidance, all updated regularly. Notably, LLM-based reasoning identified an entirely novel tax strategy, highlighting these models’ potential to revolutionize tax agencies’ fight against tax abuse. Code & Data — https://github.com/BlairStanek/LLM-Tax-Abuse/ 1 Introduction Figure 1: Diagram of a novel tax strategy generated by o1-preview during our experiments. Creating such strategies normally requires extensive effort from expensive, specialized domain experts. While we find no LLMs consistently generating workable strategies, to our knowledge this is the first time an LLM has independently invented a tax strategy. Tax-minimization schemes cost the U.S. government over $600 billion annually, a staggering sum ( IRS 2024 ) . Other governments around the world also lose substantial tax revenue to such schemes ( OECD 2025 ) . This problem is uniquely susceptible to being solved by artificial intelligence, since it involves huge amounts of data and millions of words of tax-law authorities, beyond the capacity of any one individual to master. But to fulfill this promise, AI must be able to reason over the text of tax-law authorities, understand how tax strategies work, and generate possible tax strategies based on existing tax-law authorities. We benchmark several leading LLMs’ abilities to do this; we find them generally mediocre but with occasional flashes of brilliance. We created a novel dataset called Shelter Check, crafted by hand by tax-law domain experts working from public sources that describe schemes used in the past by taxpayers to minimize taxes. The dataset includes the text of the tax-law authorities, like tax code sections or court decisions, that enabled the scheme. It also includes: the background facts necessary for a taxpayer to use the scheme; the goals of the scheme (e.g. creating an artificial tax deduction); the steps of the scheme itself; and a legal analysis of why the scheme arguably meets the goals, given the tax-law authorities. We also include an adversarial step that, if used by a taxpayer, would make the strategy unquestionably fail to meet at least one of the goals. We use Shelter Check to thoroughly test the abilities of OpenAI’s o3, Anthropic’s Claude-4, and Google’s Gemini-2.5. As an initial step, we test whether the LLMs can validate each step of our legal analysis of each strategy, evaluating whether it is correct or merely viable. More substantively, we test whether the LLMs can analyze whether the stated goals are met by the strategy steps, given the background facts and textual authorities. To test the LLMs’ abilities to identify tax strategies that do not work, we repeat this experiment, using the adversarial step that should prevent the strategy from working. We then evaluate the ability of the LLMs to generate tax strategies. We try a “step-cloze” task where we ask the LLMs to fill in a single blanked-out step in our known tax strategies. We also asked LLMs to generate entire tax strategies from scratch, prompting them with only the background, goals, and authorities behind known past strategies, then having domain experts investigate their feasibility. A domain expert graded these model-generated strategies; we also had other models grade these strategies and compared the models’ grades against the domain expert’s grades. Notably, o1-preview generated what appears to be an entirely novel working tax-minimization strategy, which is depicted in Figure 1 , discussed in section 4 , and posted on GitHub. This result suggests LLMs might have revolutionary potential for helping governments find and thus shut down such strategies. 2 Background Automated legal reasoning has a history dating back to expert systems. Legal rules can be formalized into Prolog-style expert systems that compute answers to straightforward legal questions ( McCarty 1977 ; Sergot et al. 1986 ) . The formalization makes explicit some amount of common sense knowledge and implicit assumptions, so that this step may require the involvement of a legal expert ( Merigoux, Chataing, and Protzenko 2021 ) . This rule-based reasoning can be naturally extended to case-based reasoning, provided the applicable rules (analogy, proportionality, etc.) are formalized too ( v. d. L. Gardner 1983 ; Brüninghaus and Ashley 2003 ; Gray et al. 2024 ) . Cao et al. 2024 focus on detecting tax evasion in raw data. In principle, a perfect translation of all tax authorities from natural language into computer-interpretable code would make the search for tax loopholes a matter of running a closed-world simulation. One example of such a simulation is the work of Hemberg et al. 2015 and Hemberg et al. 2016 . They model the relationship between tax evasion and auditing policies, and how each dynamically adapts to the other’s behavior. But the scalability of such efforts is unclear, given the effort required to translate just a few sections of tax law into computer-interpretable form. Fratrič, Holzenberger, and Amariles 2025 let virtual agents explore the full scope of possibilities allowed by a formal model of the law, constructed through a costly process of translating law to code. The economic profit associated with a specific sequence of agent actions is then used as an indicator for potentially illegal tax strategies. But legal authorities are written in natural language, potentially making it preferable to engage directly with the legal text. Several datasets have been built around tasks involving legal text. LexGLUE ( Chalkidis et al. 2022 ) groups multiple English-language legal NLP datasets. All tasks are classification tasks, except for one multiple-choice QA task. Analogous datasets exist for other languages and jurisdictions, typically framed as classification or retrieval tasks: judgment prediction for Indian courts ( Malik et al. 2021 ) and Swiss courts ( Niklaus, Chalkidis, and Stürmer 2021 ) , multiple choice questions from the National Judicial Examination of China ( Zhong et al. 2020 ) , and retrieval of statutes and cases for Japanese law ( Goebel et al. 2024 ) , inter alia . Closer to our dataset, LegalBench ( Guha et al. 2023 ) is a collection of legal tasks, some from existing datasets, some contributed by legal experts. The tasks expect a wide variety of output formats, and are meant to test LLM abilities. In particular, LegalBench contains two tasks that explicitly involve tax: the statutory reasoning assessment (SARA); and predicting outcomes in cases from the Canada Tax Court. Both are framed as classification tasks. While state-of-the-art LLMs reach high performance on the self-contained benchmarks cited above, Nay et al. 2024 have further documented LLM abilities closer to the practice of tax law, reporting mixed results. Similarly, Zhou et al. 2024 benchmark LLMs’ abilities to apply several rulesets, one of which is a curated subset of the basic U.S. tax

Read the complete text