Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning
Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral · 2025
Read Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning by Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-05675.txt.
Introduction
Venkatesh Mishra and co-authors dissect where language models go wrong when they reason through legal problems step by step. Their 2025 paper uses college-level civil procedure questions, where answers demand careful use of rules and precedents. Instead of grading only the final answer, the team grades every link in the chain.
From hand-checked examples they build a taxonomy of reasoning errors, scored for soundness and correctness. An automatic judge, itself a language model, then applies that taxonomy at scale across several models. The scores expose recurring weak spots that raw accuracy hides. Feeding the error list back as prompting feedback lifts performance a little. The framework doubles as a reusable tool for error analysis in other logic-heavy tasks.
Catalog record
| CREATOR | Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-05675
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating the Shortcomings of LLMs in Step-by-Step Legal Reasoning Venkatesh Mishra, Bimsara Pathiraja, Mihir Parmar, Sat Chidananda, Jayanth Srinivasa, Gaowen Liu, Ali Payani, Chitta Baral Abstract Reasoning abilities of LLMs have been a key focus in recent years. One challenging reasoning domain with interesting nuances is legal reasoning, which requires careful application of rules, and precedents while balancing deductive and analogical reasoning, and conflicts between rules. Although there have been a few works on using LLMs for legal reasoning, their focus has been on overall accuracy. In this paper, we dig deeper to do a step-by-step analysis and figure out where they commit errors. We use the college-level Multiple Choice Question-Answering (MCQA) task from the Civil Procedure dataset and propose a new error taxonomy derived from initial manual analysis of reasoning chains with respect to several LLMs, including two objective measures: soundness and correctness scores. We then develop an LLM-based automated evaluation framework to identify reasoning errors and evaluate the performance of LLMs. The computation of soundness and correctness on the dataset using the auto-evaluator framework reveals several interesting insights. Furthermore, we show that incorporating the error taxonomy as feedback in popular prompting techniques marginally increases LLM performance. Our work will also serve as an evaluation framework that can be used in detailed error analysis of reasoning chains for logic-intensive complex tasks 1 1 1 Data and source code are available at https://github.com/VenkyMishra/legal_reasoning . 1 Introduction Figure 1: An example of determining domicile in a legal context. A reasoner must discern whether the condition of ‘indefinite to stay in a place’ is met. While many LLMs predict Marla is domiciled in Montana since her program is only for 2 years, legally, her ambiguous plans indicate an intent to remain in Denver indefinitely, making her domiciled in Denver, not Montana. Figure 2: Overview of the proposed pipeline for evaluating legal reasoning in LLMs. The process begins with converting Civ. Pro. dataset ( top left ), followed by generating reasoning chains using LLMs in a zero-shot CoT setting ( bottom left ). These chains are manually analyzed for various error types ( top right ), based on the proposed error taxonomy. The pipeline is then automated using an LLM-based system ( bottom right ) to assess reasoning chains for errors such as misinterpretation, providing insights into the LLMs’ reasoning accuracy. Legal reasoning is a complex process requiring the careful application of rules, and precedents while balancing deductive and analogical reasoning, for various legal scenarios ( Walker, 2007 ; Hafner and Berman, 2002 ) . These challenges are heightened by reasoning through uncertainties (Figure 1 ) and ambiguous laws (Figure 19 ). In recent years, Large Language Models (LLMs) have emerged as the most dominant AI models to process and generate natural language. There has been widespread research to showcase the emergence of natural language understanding (NLU) and reasoning abilities ( Zelikman et al., 2022 ; Zelikman et al., 2024 ; Hao et al., 2023 ; Mondorf and Plank, 2024 ; Lanchantin et al., 2024 ) of LLMs. These findings have opened up avenues to utilize LLMs in complex domains like Law ( Lai et al., 2023 ) . Consequently, there has been significant research in evaluating the performance of LLMs’ legal reasoning ability ( Guha et al., 2023 ; Blair-Stanek et al., 2023 ; Kang et al., 2023 ) . In this paper, we focus on providing a detailed, fine-grained analysis of the errors that occur during step-by-step legal reasoning using LLMs. While earlier works exist on evaluating step-by-step reasoning of LLMs ( Golovneva et al., 2023 ; Prasad et al., 2023 ) , they do not specifically cater to legal reasoning. As shown in Figure 1 , analyzing legal scenarios requires extensive consideration of critical analysis of prior context. Hence, beyond just evaluating final answers, it is crucial to analyze the step-by-step reasoning chains generated by LLMs and where they falter to gauge their reasoning capabilities to solve such tasks. Despite continued improvements, LLM outputs are affected by fundamental challenges such as hallucinations ( Dahl et al., 2024 ; Varshney et al., 2024 ) , and misunderstanding long-contexts ( Lu et al., 2024 ) . Our work aims to investigate such errors on a fine-grained level for each step generated to showcase the ability of LLM to perform legal reasoning. To this end, we leveraged the dataset from Bongard et al. (2022) (referred to as ‘ Civ. Pro. ’ throughout the paper) in the form of MCQA and analyzed reasoning chains generated by LLMs. The dataset comprises of 175 legal scenarios, each providing a comprehensive legal context that includes relevant rules, precedents, and exceptions. Following this context, questions and options are presented to test the understanding and application of these legal principles. The primary goal of using this dataset is to assess how well the LLM reasons logically and contextually when provided with all necessary legal rules, without relying on its inherent knowledge to recall them. This approach highlights the LLM’s ability to perform deductive and analogical reasoning in complex, context-rich legal scenarios. Some of the widely used LLMs, Mistral-7B-v2-Instruct, Llama-3-8B-Instruct, GPT-3.5-turbo, GPT-4-turbo, and GPT-4o, are evaluated in zero-shot-CoT setting ( Wei et al., 2023 ) (as shown in Figure 2 ). Subsequently, human evaluation of the LLM-generated reasoning chains against the ground truth solutions is performed for a subset of the dataset to find the most common errors being committed in the reasoning steps. This resulted in the development of a detailed error taxonomy, as outlined in Tables 1 and 2 , which allowed for a deeper understanding of the underlying causes of LLMs’ failures. Additionally, to extend this evaluation across the entire dataset, we develop a method employing ‘LLMs as auto-evaluators’ (inspired by Liu et al. (2023) ; Chern et al. (2024) ) that leverages our proposed error taxonomy. The auto-evaluator framework achieves a recall of \sim 87.06 \% in terms of identifying errors similar to human annotation. To derive better qualitative observations, we utilize two metrics: soundness and correctness (details in § 4.2 ) to further analyze the reasoning chains. These metrics provide us with interesting insights, the most prominent of them being the inability to generate error-free rationales due to misinterpretation of the contextual nuances. In the end, we apply various prompting strategies (details in § 5.3 ) known to enhance LLM reasoning abilities with the zero-shot method. We also incorporated our error taxonomy as feedback to the prompting strategies to assess their effectiveness in mitigating errors in legal reasoning. The prompting strategies enhanced with error taxonomy feedback show improved accuracy (maximum of \sim 4 \% ) which suggests that providing information about errors help in improving the performance of LLMs. We hope that this framework for automatically evaluating step-by-step reasoning in complex tasks will be helpful for future research. In summary, the main contributions of our paper are: 1. We propose an error taxonomy to systematically identify the most commonly occurring errors in step-by-step legal reasoning. 2. We develop an LLM-based pipeline to automatically detect errors, and introduce two key metrics—soundness and correctness—to evaluate step-by-step legal reasoning. 3. We investigate the integration of error-taxonomy feedback into various LLM prompting strategies and find that it enhances the