Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks

Antoine Bigeard, Rayan Krishnan, Shirley Wu, Langston Nashold · 2025

Read Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks by Antoine Bigeard, Rayan Krishnan, Shirley Wu, Langston Nashold (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-00828.txt.

Introduction

Antoine Bigeard and co-authors set in 2025 a hard test of chat agents on true cash world research tasks. Cash work is a prime field for such agents, yet their full reach there is not proved. The test draws tasks from fresh firm files.

Nine task kinds, drawn with bank and fund pros, frame five hundred plus pro writ asks. Each ask passes a strict check for truth and fit. Agents get web search and a firm file base to work with. The best tool scores past mid at a few coins per ask. High stakes use still waits on more growth.

Catalog record

CREATORAntoine Bigeard, Rayan Krishnan, Shirley Wu, Langston Nashold
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-00828

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Finance Agent Benchmark: Benchmarking LLMs on Real-world Financial Research Tasks Antoine Bigeard, Rayan Krishnan, Shirley Wu, Langston Nashold Abstract Artificial Intelligence (AI) technology has emerged as a transformative force in financial analysis and the finance industry, though significant questions remain about the full capabilities of Large Language Model (LLM) agents in this domain. We present the Finance Agent Benchmark, featuring challenging and diverse real-world finance research problems which require LLMs to perform complex analysis with the use of of recent SEC filings. We construct the benchmark using a taxonomy of nine financial task categories, developed in consultation with experts from banks, hedge funds, and private equity firms. The dataset includes 537 expert-authored questions, covering tasks from information retrieval to complex financial modeling, where each question was validated through a rigorous review process to ensure accuracy and relevance. Moreover, we implement an agentic harness that equips LLMs with tools sufficient to produce an accurate response, including as Google Search and EDGAR database access. Overall, Finance Agent Benchmark provides a comprehensive testbed for measuring the progress of LLM-driven finance agents. Our evaluation reveals significant limitations in current AI capabilities—even the best-performing model (OpenAI’s o3) achieved only 46.8% accuracy, at an average cost of $3.79 per query. It underscores the need for further advancements before reliable deployment in high-stakes finance settings. \displaystyle 0.0 \displaystyle 0.5 \displaystyle 1.0 \displaystyle 1.5 \displaystyle 2.0 \displaystyle 2.5 \displaystyle 3.0 \displaystyle 3.5 \displaystyle 4.0 0 20 40 Cost per Question (USD) Accuracy (%) o3 Claude 3.7 Sonnet o4 Mini Grok 3 Mini Fast High Reasoning Gemini 2.5 Pro Preview Grok 3 Beta GPT 4.1 mini Mistral Small 3.1 (03/2025) Llama 4 Maverick Figure 1 : Cost-Accuracy pareto curve results on Finance Agent Benchmark. The Finance Agent Benchmark reveals a clear logarithmic relationship between accuracy and cost, with a sharp diminishing return beyond $1 USD per question—highlighting that even today’s most sophisticated models struggle to achieve greater than 50% accuracy on real-world financial tasks. 1 Introduction The finance sector is a critical domain with large economic impact and operational scale. With daily foreign exchange turnover averaging $7.5 trillion bis2022fxturnover , financial operations demand extensive human resources for routine tasks. Entry-level finance professionals sometimes spend up to 40% of their workweek on gathering data rather than analyzing it pwc_effectiveness_benchmark —time that could be better used for strategic analysis that drives business value. The repetitive and essential nature of these tasks requires significant workforce investment and creates inefficiencies. Recent developments in autonomous AI agents offer a promising solution for financial workflows yehudai2025surveyevaluationllmbasedagents ; kapoor2024aiagents . These systems have shown advancements in their ability to perform complex tasks that involve unstructured data and multi-step reasoning—capabilities essential for financial analysis an2024finverseautonomousagentversatile . By automating routine yet time-consuming tasks, AI agents can greatly reduce human effort. However, despite this clear potential, there remains a need for robust, domain-specific benchmarks to assess the capabilities of financial agents. Previous benchmarks often fail to address the interactive environments and nuanced reasoning an agent must undergo when completing industry-specific tasks, leaving performance in real-world scenarios uncertain chen2025standardbenchmarks . This deficiency is particularly concerning given that AI models can generate plausible-sounding misinformation and hallucinations williamson2024hallucinations , which, if relied upon for high-stake, high-impact decisions, could lead to severe negative outcomes. For instance, if an AI model misinterprets a company’s earnings report, incorrectly identifying a consistent earnings beat over four consecutive quarters, it could lead to misguided decision to invest in the company. Without proper evaluation, these systems cannot be trusted for high-stakes financial applications. A dedicated benchmark is essential for tracking the state-of-the-art financial agents and highlighting the remaining deficits in model capabilities in this domain. Figure 2 : Architecture of the Finance Agent Benchmark. The framework features a structured evaluation process with four key steps: (1) Data Creation: experts identify practical and common financial questions requiring access to public financial documents and provide reference answers. (2) Rubric Development: expert-generated data is used to create robust rubrics with expected calculations and reasoning steps for standardized LLM evaluation. (3) Agent Evaluation: questions are processed through LLMs equipped with necessary tools to generate answers. (4) Answer Grading: an LLM-as-judge scoring system using LLM-as-judge applies conjunction rules to determine correctness across multiple criteria. To fill this gap, we present Finance Agent Benchmark (Figure 2 ), a novel, standardized framework to rigorously evaluate AI agents on real-world financial analysis tasks. Our benchmark offers: • Realistic queries representing real-world challenges: We collaborated with seven domain experts across the financial services industry (banks, hedge funds, private equity) to identify and categorize common analytical tasks performed with public financial documents, see Table 1 . • Step-by-step expert annotation for reliable validation : We constructed this comprehensive dataset with questions, corresponding answers and reasoning trajectories (steps to answer) created by the experts. Each question is verifiable through public U.S. Securities and Exchange Commission (SEC)’s filings from the Electronic Data Gathering, Analysis, and Retrieval (EDGAR) database, official repository for public company filings, and underwent peer-review by the experts. • Multi-dimensional Evaluation Framework : Our benchmark uses an LLM-as-judge approach with rubric-based assessment that evaluates answers against specific components of expert solutions rather than holistically. The framework incorporates multiple accuracy verification checks and a contradiction detection mechanism to identify factual inconsistencies between generated and expert answers. • Model-Agnostic Evaluation Harness for Benchmark Baseline : We developed an agentic evaluation framework (Figure 3 ) equipped with multiple tools, including Google Search and EDGAR Research Filing Search, enabling standard benchmarking and performance assessment across different LLMs. We established performance baselines using our evaluation harness, providing a foundation for future research in finance-focused AI agents. Our analysis in Figure 1 indicates that substantial progress is still needed—the best-performing baseline model, o3, achieved only 46.8% accuracy. This highlights the benchmark’s value in testing the ability of models to perform time-intensive tasks expected of entry level finance analysts. Notably, no model surpassed 50% accuracy, underscoring the advancements required before these systems can be deployed autonomously and reliably in the financial sector. Nevertheless, these models demonstrate notable efficiency advantages: even the most expensive model (o3) averages just 3.1 minutes per task and costs $3.78, compared to human experts requiring 16.8 minutes and costing $25.66 1 1 1 These calculations only account for time spent directly answering the question, and exclude the time to create the question, review the answer, etc. for equivalent analyses. This efficiency-to-performance ratio points to the promising potential of these models in supporting human analysts, even as work continues toward improving their accuracy for more autonomous applications. 2 Finance Agent Benchmark: High-quality Financial Benchmark 2.1 Dataset Creation Process Expert Consultation and Taxonomy Development. We collaborated with financial industry experts from banks, hedge funds, and startups to identify common analytical tasks and develop a comprehensive taxonomy of finance analysis questions. Figure 2 shows the overall development process of the benchmark. The experts had at least 2-3 years of working experience at bulge bracket firms such as Goldman Sachs and J.P. Morgan. The resulting taxonomy categorizes tasks based on complexity and frequency in real investment banking contexts, ranging from basic retrieval to complex market analysis. See dataset taxonomy in Table 1 . Question Generation and Quality Control. The experts then crafted 537 questions across

Read the complete text