FinChain : A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning
Zhuohan Xie Dhruv Sahnan Debopriyo Banerjee Georgi Georgiev, Rushil Thareja Hachem Madmoun Jinyan Su Aaryamonvikram Singh, Yuxia Wang Rui Xing Fajri Koto Haonan Li Ivan Koychev, Tanmoy Chakraborty Salem Lahlou Veselin Stoyanov Preslav Nakov · 2025
Read FinChain : A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning by Zhuohan Xie Dhruv Sahnan Debopriyo Banerjee Georgi Georgiev, Rushil Thareja Hachem Madmoun Jinyan Su Aaryamonvikram Singh, Yuxia Wang Rui Xing Fajri Koto Haonan Li Ivan Koychev, Tanmoy Chakraborty Salem Lahlou Veselin Stoyanov Preslav Nakov (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-02515.txt.
Introduction
Zhuohan Xie and a large team introduce FinChain, a benchmark for verifiable chain-of-thought reasoning in finance, in work from 2025. Older finance datasets grade only final numeric answers, which hides the steps that make an answer checkable. FinChain covers 58 topics across 12 financial domains, each built from symbolic templates with runnable Python code.
A new measure, CHAINEVAL, grades both final answers and step-level consistency. Tests of 26 leading language models found clear limits in symbolic financial reasoning, even for top models. Models tuned for the domain or for math narrowed the gap a lot. The authors offer the benchmark as a base for trustworthy financial AI.
Catalog record
| CREATOR | Zhuohan Xie Dhruv Sahnan Debopriyo Banerjee Georgi Georgiev, Rushil Thareja Hachem Madmoun Jinyan Su Aaryamonvikram Singh, Yuxia Wang Rui Xing Fajri Koto Haonan Li Ivan Koychev, Tanmoy Chakraborty Salem Lahlou Veselin Stoyanov Preslav Nakov |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-02515
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FinChain : A Symbolic Benchmark for Verifiable Chain-of-Thought Financial Reasoning Zhuohan Xie Dhruv Sahnan Debopriyo Banerjee Georgi Georgiev, Rushil Thareja Hachem Madmoun Jinyan Su Aaryamonvikram Singh, Yuxia Wang Rui Xing Fajri Koto Haonan Li Ivan Koychev, Tanmoy Chakraborty Salem Lahlou Veselin Stoyanov Preslav Nakov Abstract Multi-step symbolic reasoning is critical for advancing downstream performance on financial tasks. Yet, benchmarks for systematically evaluating this capability are lacking. Existing datasets like FinQA and ConvFinQA supervise only final numerical answers, without assessing intermediate reasoning steps. To address this, we introduce FinChain , the first symbolic benchmark designed for verifiable Chain-of-Thought (CoT) financial reasoning. Spanning 54 topics across 12 financial domains, FinChain offers five parameterized templates per topic, each varying in reasoning complexity and domain expertise required. Each dataset instance includes an executable Python trace, enabling automatic generation of extensive training data and easy adaptation to other domains. We also introduce ChainEval , a new metric for automatic evaluation of both final answers and intermediate reasoning. Benchmarking 30 LLMs on our dataset, we find that even state-of-the-art models have considerable room for improvement in multi-step financial reasoning. All templates and evaluation metrics for FinChain are available at https://github.com/mbzuai-nlp/finchain . 1 Introduction Large language models (LLMs) have demonstrated strong performance across a wide range of tasks ( Zhao et al. 2023 ; Xie et al. 2023b ) , including practical applications in fields such as finance, healthcare and law ( Chen et al. 2024b ) . In the financial field, for instance, effective analysis often requires processing large volumes of text from diverse sources, including documents, news articles, and social media. These texts frequently reflect or shape financial phenomena, such as market sentiment ( Nie et al. 2024 ) . FinChain Template (Compound Interest) #Question: investor_name invested principal in project_name . The investment grows at an annual interest rate of rate % compounded annually over time years. Calculate the compound interest (CI) . #Variables: - investor_name = sample(investors) - project_name = sample(projects) - principal = range(1000, 5000) - rate = uniform(2, 10) - time = range(1, 5) #Chain-of-Thought Solution: Step 1: Compute the compound amount: amount = principal \times \left(1+\frac{\hbox{\pagecolor{varrate}{rate}}}{100}\right)^{\hbox{\pagecolor{vartime}{time}}} Step 2: Compute the compound interest: CI = amount - P Figure 1: Symbolic template for generating compound interest problems in FinChain . Prior work in financial NLP has predominantly focused on adapting general-purpose tasks to financial text. These tasks include information extraction ( Shah et al. 2023 ) , sentiment analysis ( Pei et al. 2022 ) , and text classification ( Sy et al. 2023 ) , where models are trained to output relatively simple targets such as entity spans, sentiment labels, or numerical scores. In contrast, our task addresses financial reasoning, which requires models to generate complete chain-of-thought (CoT) reasoning traces that justify each step in solving a financial problem, as shown in Figure 1 . Existing benchmarks for financial reasoning, such as FinQA ( Chen et al. 2021 ) and ConvFinQA ( Chen et al. 2022 ) , frame the task primarily as numerical question answering. These datasets focus on predicting the final answer and do not require models to explicitly generate or justify intermediate reasoning steps. While some examples may include traces of intermediate operations (loosely resembling reasoning steps), these are neither comprehensive nor rigorously structured to reflect the standards of financial analysis. In particular, they lack explicit, verifiable representations of a full reasoning chain, which is critical for transparency and accountability in financial decision-making. As a result, these benchmarks offer limited diagnostic value: they do not reliably reveal where a model’s reasoning succeeds or fails, nor do they distinguish genuine multi-step reasoning from shallow pattern matching. In the field of mathematical reasoning, the symbolic template paradigm introduced by GSM-Symbolic ( Mirzadeh et al. 2024 ) has become a widely adopted approach. This method re-templates 100 math problems from GSM8K ( Cobbe et al. 2021 ) , providing explicit intermediate steps alongside the final answer. We adopt a similar templating strategy, but construct our dataset entirely from scratch for the financial field. As illustrated in Figure 1 , each symbolic template encodes a parameterized financial problem (e.g., compound interest) with variable named entities and numerical inputs. Each template is paired with executable Python code that computes both intermediate steps and the final result. This design enables scalable, contamination-free generation of consistent examples for both training and evaluation. The financial field comprises a wide range of distinct domains, each involving different stakeholders and downstream objectives. Consequently, it is essential to design the dataset in a templated manner that supports domain-specific evaluation. To address this, we organize our dataset using a fine-grained financial taxonomy ( Figure 2 ) covering 12 high-level domains (e.g., Corporate Finance, Sustainable Finance, Crypto), each comprising distinct topics, 54 topics in total . For each topic, we develop five parameterized templates (two easy, two intermediate, one advanced), varying in symbolic reasoning complexity and required domain knowledge. To the best of our knowledge, this hierarchical structure constitutes the most detailed classification of financial reasoning tasks among existing datasets in this area. Each templated instance includes: (1) a scenario card describing the topic, difficulty tier, and sampled inputs (e.g., “Discounted cash flow valuation, advanced”), and (2) an executable chain of reasoning steps in Python, grounded in domain-specific equations and concepts. Because every operation in the chain is explicit and executable, our benchmark supports full machine-verifiability: any hallucinated, skipped, or incorrect step in a model’s output can be automatically detected. This contrasts sharply with existing datasets like FinQA and ConvFinQA , which only supervise the final answer. To enable rigorous evaluation, we introduce ChainEval , which assesses both the correctness of the final answer and the alignment of intermediate reasoning steps. Our evaluation across proprietary and open-source LLMs reveals a consistent pattern: while large models such as GPT-4.1 and LLaMA 3.3 (70B) achieve high overall accuracy, they still struggle with complex symbolic tasks and multi-step financial reasoning. In contrast, smaller models consistently underperform, even when fine-tuned on financial data. These findings indicate that domain-specific supervision alone is insufficient; robust reasoning performance also depends on adequate model capacity for handling symbolic and multi-hop inference. Our main contributions in this study are summarized below. • We present the first from-scratch symbolic benchmark for financial reasoning, constructed using a custom designed financial taxonomy spanning 12 domains and 54 topics. For each topic, we provide five parameterized templates (two easy, two intermediate, one advanced) along with executable Python code that generates step-by-step reasoning traces for each instance. • We propose an automatic alignment metric that evaluates a model’s chain of thought solution by comparing it against gold-standard reasoning traces, assessing both the correctness of the final answer and the alignment of intermediate steps. • We evaluate a range of open-source and proprietary large language models and large reasoning models, and observe that even state-of-the-art models exhibit substantial difficulty in handling symbolic financial reasoning—particularly on advanced templates that require longer and more complex reasoning chains. Figure 2: FinChain taxonomy of financial reasoning topics. Our benchmark spans 54 topics organized into 12 major domains, ranging from traditional areas like Corporate Finance and Financial Reporting to emerging fields such as Crypto Finance and Sustainable Finance . This hierarchical