FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles

Arun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang, Vivek Gupta, Qingyun Wang, Yanjie Fu, Denghui Zhang · 2026

Read FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles by Arun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang, Vivek Gupta, Qingyun Wang, Yanjie Fu, Denghui Zhang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11339.txt.

Introduction

Arun Vignesh Malarkkan and coauthors present FinRule-Bench, a benchmark testing whether language models can audit real financial statements against explicit accounting principles. Earlier benchmarks leave this skill unclear. Four statement types pair with human-curated rules across three tasks: verifying one rule, naming the rule that was violated, and diagnosing several violations at once down to the record. Models handle single-rule checks well. They fall sharply on discrimination and multi-violation diagnosis. This exposes weak spots in high-stakes reasoning.

Catalog record

CREATORArun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang, Vivek Gupta, Qingyun Wang, Yanjie Fu, Denghui Zhang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86965
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.11339

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FinRule-Bench: A Benchmark for Joint Reasoning over Financial Tables and Principles Arun Vignesh Malarkkan, Manan Roy Choudhury, Guangwei Zhang, Vivek Gupta, Qingyun Wang, Yanjie Fu, Denghui Zhang Abstract . Large language models (LLMs) are increasingly applied to financial analysis, yet their ability to audit structured financial statements under explicit accounting principles remains poorly explored. Existing benchmarks primarily evaluate question answering, numerical reasoning, or anomaly detection on synthetically corrupted data, making it unclear whether models can reliably verify or localize rule compliance on correct financial statements. We introduce FinRule-Bench , a benchmark for evaluating diagnostic completeness in rule-based financial reasoning over real-world financial tables. FinRule-Bench pairs ground-truth financial statements with explicit, human-curated accounting principles and spans four canonical statement types: Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark defines three auditing tasks that require progressively stronger reasoning capabilities: (i) rule verification , which tests compliance with a single principle; (ii) rule identification , which requires selecting the violated principle from a provided rule set; and (iii) joint rule diagnosis , which requires detecting and localizing multiple simultaneous violations at the record level. We evaluate LLMs under zero-shot and few-shot prompting, and introduce a causal-counterfactual reasoning protocol that enforces consistency between decisions, explanations, and counterfactual judgments. Across tasks and statement types, we find that while models perform well on isolated rule verification, performance degrades sharply for rule discrimination and multi-violation diagnosis. Errors are dominated by incomplete coverage and mislocalization of violations, revealing systematic failures that are not captured by existing financial or tabular benchmarks. FinRule-Bench provides a principled and reproducible testbed for studying rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis. We release the FinRule-Bench benchmark along with the complete evaluation codebase, including task-specific LLM prompting scripts for all three tasks across all financial statement types, at https://anonymous.4open.science/r/FinRule-Bench-6723/ . 1. Introduction Large language models (LLMs) are increasingly deployed for financial analysis and decision support, including applications such as numerical question answering, report summarization, and anomaly detection. However, a central task in financial auditing remains poorly understood from the perspective of LLM evaluation: determining whether structured financial statements comply with explicit accounting principles and, if not, identifying and localizing the precise sources of non-compliance. This requirement goes beyond producing correct answers or detecting isolated inconsistencies; it demands diagnostic completeness , i.e., the ability to exhaustively enforce a formal rule system over structured, interdependent financial tables. Financial reporting underpins modern capital markets and serves as the primary mechanism through which firms communicate performance, risk, and regulatory compliance to investors and oversight bodies. The reliability of these reports is enforced through rigorous auditing processes overseen by regulatory authorities such as the PCAOB ( 34 ) and grounded in formal accounting frameworks including U.S. GAAP ( 12 ) and IFRS ( 20 ) . Auditing requires verifying that financial statements satisfy a large set of explicit and interdependent accounting principles, including arithmetic identities, hierarchical aggregation constraints, and conditional applicability rules. Effective auditing therefore requires structured reasoning that can be evaluated for completeness, requiring models to identify the violated principles and to attribute each violation localized to the specific records, rather than producing binary judgments or isolated answers. Figure 1 . FinRule-Bench Benchmark Overview. FinRule-Bench evaluates LLMs on financial auditing tasks that require both intra-table reasoning and joint reasoning across multiple financial statements under explicit accounting rules. Existing benchmarks only partially capture these requirements. General table question answering datasets, such as TAT-QA ( 44 ) , primarily evaluate single-table extraction and numerical reasoning over individual tables, without requiring validation against external rule systems. Financial question answering benchmark,s including FinQA ( 2 ) and ConvFinQA ( 3 ) , emphasize retrieval and arithmetic computation but do not assess whether models can enforce accounting principles or discriminate among competing rule violations. As a result, models can achieve high accuracy on existing benchmarks while remaining unsuitable for auditing, where correctness depends on exhaustive rule enforcement and precise attribution rather than isolated answers. This gap motivates a distinct evaluation setting. Verifying accounting compliance over ground-truth financial statements differs fundamentally from question answering or anomaly detection. In realistic auditing scenarios, financial tables are often correct by construction, and failures arise from incorrect reasoning rather than from corrupted or noisy data. Consequently, evaluating LLMs in this setting requires benchmarks that isolate reasoning fidelity from data artifacts and directly test whether models can exhaustively identify violated rules and attribute each violation to the appropriate records in structured financial reports. To address this need, we introduce FinRule-Bench , a benchmark designed to evaluate rule-based financial reasoning over real-world financial statements. FinRule-Bench pairs structured financial tables with explicit, human-curated accounting principles and defines three auditing tasks that require progressively stronger reasoning capabilities: rule verification, which tests compliance with a single principle; rule identification, which requires selecting the uniquely violated principle from a provided rule set; and joint rule diagnosis, which requires detecting and localizing multiple simultaneous violations at the record level. Together, these tasks evaluate a model’s ability to compose accounting rules, distinguish violated principles, and localize violations to specific records under realistic auditing conditions. Beyond task formulation, FinRule-Bench introduces a causal-counterfactual reasoning protocol for structured financial analysis. This protocol requires models not only to output decisions, but also to justify them through explicit causal explanations and to assess whether minimal counterfactual modifications to the financial table would alter those decisions. Rather than aiming to improve accuracy uniformly, this protocol serves as a diagnostic instrument: by comparing decisions and counterfactual judgments, it reveals inconsistencies, incomplete coverage, and mislocalization errors that are not observable when evaluating final answers alone. Overall, FinRule-Bench provides a principled testbed for evaluating rule-governed reasoning, diagnostic coverage, and failure modes of LLMs in high-stakes financial analysis. By focusing on rule compliance over correct financial statements, it complements existing tabular and financial benchmarks and enables precise analysis of where current models succeed and where they fundamentally fall short. We summarize our contributions as follows: (1) Benchmark. We introduce FinRule-Bench , a benchmark derived from real-world financial statements spanning Balance Sheets, Cash Flow Statements, Income Statements, and Statements of Equity. The benchmark pairs ground-truth tables with explicit accounting principles, enabling controlled evaluation of rule-based financial reasoning beyond question answering or anomaly detection. (2) Task Suite. We formalize three complementary auditing tasks that evaluate progressively stronger reasoning capabilities: rule verification, which tests compliance with a single accounting principle; rule identification, which requires selecting the violated principle from a provided rule set; and joint rule diagnosis, which requires multi-violation detection and record-level localization. (3) Reasoning and Evaluation Framework. We introduce a causal-counterfactual prompting protocol for structured financial reasoning and introduce diagnostic evaluation analyses, including rule-complexity breakdowns, error-type categorization, and performance-cost scalability studies, that reveal systematic failures in diagnostic completeness not captured by aggregate accuracy alone. 2. Related Work LLMs for Table Understanding. Applying LLMs over tabular data poses challenges that differ fundamentally from textual reasoning, including modeling two-dimensional structure, performing precise numerical reasoning, and preserving semantic relationships across rows and columns ( 11 ) . Early work addressed these challenges through task-specific, encoder-based models such as TaPas ( 17 ) , TaBERT ( 41 ) , TURL ( 7 ) , and TUTA ( 37 ) ,

Read the complete text