FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs
Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Ruiyu Wang, Bo Li, Xiao Huang, Dongning Sun, Xinrun Wang · 2025
Read FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs by Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Ruiyu Wang, Bo Li, Xiao Huang, Dongning Sun, Xinrun Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13533.txt.
Introduction
Junzhe Jiang and co-authors present a test suite for language models in finance work in this 2025 paper. Finance tasks are slow to do by hand, allow almost no errors, and pull data from many split sources. Past test suites for models in this field used thin data and simple tasks, the authors argue.
Their suite has three parts. One part builds fake but lawful company data that mimics real markets. A second part sets close to two hundred tasks across money skills, audits, and advice work. A third part scores all runs in one shared way. Top models did well on basic tasks but fell steeply on hard, multi-step ones. Small math slips grew as tasks chained together.
Catalog record
| CREATOR | Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Ruiyu Wang, Bo Li, Xiao Huang, Dongning Sun, Xinrun Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 153 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13533
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FinMaster: A Holistic Benchmark for Mastering Full-Pipeline Financial Workflows with LLMs Junzhe Jiang, Chang Yang, Aixin Cui, Sihan Jin, Ruiyu Wang, Bo Li, Xiao Huang, Dongning Sun, Xinrun Wang Abstract Financial tasks are pivotal to global economic stability; however, their efficient execution faces persistent challenges, including labor intensive processes, low error tolerance, data fragmentation, and limitations in existing technological tools. Although large language models (LLMs) have shown remarkable success in various natural language processing (NLP) tasks and have shown potential in automating workflows through reasoning and contextual understanding, current benchmarks for evaluating LLMs in finance suffer from insufficient domain-specific data simplistic task design, and incomplete evaluation frameworks. To address these gaps, this article presents FinMaster , a comprehensive financial benchmark designed to systematically assess the capabilities of LLM in financial literacy, accounting, auditing, and consulting. Specifically, FinMaster comprises three main modules: i) FinSim , which builds simulators that can generate synthetic, privacy-compliant financial data sets for different types of companies to replicate real-world market dynamics; ii) FinSuite , which provides a variety of tasks in core financial domains, spanning 183 tasks of various types and difficulty levels; and iii) FinEval , which develops a unified interface for streamlined evaluation. Extensive experiments over state-of-the-art LLMs, including GPT-4o-mini, Claude-3.7-Sonnet, and DeepSeek-V3, reveal critical capability gaps in financial reasoning, with accuracy dropping from over 90% on basic tasks to merely 40% on complex scenarios requiring multi-step reasoning. This degradation exhibits the propagation of computational errors, where single-metric calculations initially demonstrating 58% accuracy decreased to 37% in multimetric scenarios. To the best of our knowledge, FinMaster is the first benchmark that comprehensively covers full-pipeline financial workflows with challenging and realistic tasks. We hope that FinMaster can bridge the gap between the research community and industry practitioners, driving the adoption of LLMs in real-world financial practices to enhance both efficiency and accuracy. 1 Introduction Finance serves as the cornerstone of the global economic system, where financial tasks are critical for ensuring accuracy, compliance, and efficiency in economic activities, e.g., capital allocation, risk management, and investment strategic decision-making. However, their execution faces significant challenges: i) labor intensive: traditional financial tasks such as accounting and auditing rely heavily on manual operations, and financial professionals require extensive training to master complex regulations, which are repetitive and time-consuming; ii) low error tolerance: minor mistakes in financial statements, e.g., decimal errors or misclassifications, can trigger compliance risks or market volatility; iii) data fragmentation: financial data always originates from diverse sources, each with unique data structures and update frequencies. However, the latency in real-time scenarios and the poor compatibility between systems in integrating heterogeneous data may lead to data silos; iv) limitations of technological tools: existing fintech tools and rule-based systems often exhibit limitations in interpreting implicit logic in financial data and adapting to evolving regulations, making them difficult to handle complex causal reasoning tasks. The advancements in large language models (LLMs), e.g., GPT-4 [ 1 ] and DeepSeek-v3 [ 12 ] , have demonstrated remarkable successes in tasks requiring reasoning, contextual understanding, and multi-step problem-solving across various domains (e.g., code generation and math reasoning). The general-purpose capabilities of LLMs make them well-suited for automating financial workflows, reducing human effort, and minimizing errors. Table 1 : Comparison of Financial Benchmarks Benchmark Complex tasks Infinite data Holistic evaluation FinQA [ 6 ] ✗ ✗ ✗ PIXIU [ 22 ] ✓ ✗ ✗ FinanceBench [ 10 ] ✗ ✗ ✗ FinBen [ 21 ] ✓ ✗ ✓ FinEval [ 26 ] ✓ ✗ ✗ SECQUE [ 23 ] ✓ ✗ ✓ FinMaster ✓ ✓ ✓ Several recent attempts that apply LLMs to finance have demonstrated promising capabilities. FinQA [ 6 ] introduces complex numerical reasoning over financial reports but is constrained by static data and structured formats. FinBen [ 21 ] offers broader evaluation across diverse financial tasks but lacks granularity for domain-specific reasoning nuances. FinTSB [ 9 ] specializes in time-series forecasting but neglects external factors. Other benchmarks like PIXIU [ 22 ] , FinanceBench [ 10 ] , and BizBench [ 11 ] demonstrate narrow scope, primarily focusing on conventional financial NLP tasks while overlooking complex financial reasoning and real-world applications. SECQUE [ 23 ] advances LLMs evaluation by focusing on practical financial tasks requiring multi-step reasoning, but still relies on a static dataset and fails to fully capture the reality of financial workflows. These limitations highlight the need for more comprehensive benchmarks that can address the dynamic nature of financial markets, complex reasoning requirements, and practical application scenarios. Figure 1 : The three main modules of FinMaster . To address these issues, we present FinMaster , a holistic benchmark for mastering full-pipeline financial workflows with LLMs. Specifically, FinMaster has three main modules: i) FinSim , which builds a simulator that can automatically generate financial data, including transaction records and financial statements, for financial literacy, accounting, auditing and consulting. It can produce infinite synthetic datasets for various types of companies to replicate real-world market dynamics, addressing the issue of lacking of financial data due to privacy constraints. ii) FinSuite , which provides a task suite, incorporating various types of tasks in the fields of accounting, auditing, and consulting, to comprehensively evaluate the performance and capabilities of LLMs in different financial tasks. iii) FinEval , which develops a unified interface, i.e., a streamlined framework, for systematic evaluation of LLMs and provides a comprehensive evaluation that quantifies the performances of online LLMs across different tasks with in-depth analysis such as accuracy and the number of tokens. Extensive experiments over widely-used LLMs, e.g., GPT-4o-mini, Claude-3.7-Sonnet, DeepSeek-V3, and OpenAI o3-mini, show the systemic gaps in financial reasoning capabilities. While these models achieve over 90% accuracy on basic tasks, their performance drops sharply to just 40% in complex scenarios that require multi-step reasoning. Moreover, general-purpose LLMs lack domain-specific knowledge, leading to hallucinated conclusions or statistically invalid outputs when professional judgment is needed. To the best of our knowledge, FinMaster is the first financial benchmark that simulates multi-step financial operations for LLMs, serving as the fundamental testbed for the advanced LLMs. The code is released at https://github.com/JiangInsight/Finmaster and the leaderboard is shown at https://jianginsight.github.io/Finmaster_leaderboard/ . 2 Preliminaries Large Language Models. Large Language Models (LLMs) are autoregressive models that learn from textual data through unsupervised learning, modeling the joint probability of variable-length token sequences by decomposing it into a product of conditional probabilities. The factorization is expressed as: P(x)=\prod\nolimits^{n}_{i=1}P(s_{i}|s_{1},...,s_{i-1}) , where (s_{1},s_{2},...,s_{n}) represents a sequence of tokens. Trained on vast corpora and parameterized by billions of weights, LLMs encode extensive common knowledge, enabling them to generalize robustly across diverse NLP tasks with simple prompting and in-context learning, and without task-specific fine-tuning [ 20 , 14 ] . Transactions and Financial Statements. Transactions refer to the economic event or business activity that result in measurable changes in a company’s financial position and are systematically recorded to populate accurate, reliable financial statements. There are three key financial statements that form the bedrock of financial reporting and analysis: i) income statement, which dynamically reports a company’s revenues, expenses, and profits/losses over a specific period; ii) balance sheet, which provides a snapshot of a company’s financial position at a specific moment; and iii) cash flow statement, which tracks the actual cash movements,