FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets

Jan Ravnik, Matjaž Ličen, Felix Bührmann, Bithiah Yuan, Felix Stinson, Tanvi Singh · 2026

Read FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets by Jan Ravnik, Matjaž Ličen, Felix Bührmann, Bithiah Yuan, Felix Stinson, Tanvi Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-07316.txt.

Introduction

Ravnik and colleagues benchmark language models on financial spreadsheets, where private equity data is too confidential for public test sets. Their synthetic FinSheet-Bench mimics real fund structures with tangled layouts, dividers and multi-line headers. Across ten model setups, no standalone model proves accurate enough for unsupervised professional use. The best model leads overall, yet accuracy collapses on the largest spreadsheet, a pattern shared by every model tested.

Catalog record

CREATORJan Ravnik, Matjaž Ličen, Felix Bührmann, Bithiah Yuan, Felix Stinson, Tanvi Singh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62454
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.07316

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FinSheet-Bench: From Simple Lookups to Complex Reasoning, Where LLMs Break on Financial Spreadsheets Jan Ravnik, Matjaž Ličen, Felix Bührmann, Bithiah Yuan, Felix Stinson, Tanvi Singh Abstract While Large Language Models (LLMs) can accelerate text-heavy tasks in alternative investment due diligence, a gap remains in their ability to accurately extract and reason over structured tabular data from complex financial spreadsheets. Progress is held back by the lack of real industry fund portfolio datasets for benchmarking, as private equity data rooms are confidential. To address this, we introduce FinSheet-Bench , a benchmark of synthetic financial portfolio data modeled on real private equity fund structures, designed to evaluate LLM performance on text-serialized spreadsheet question answering and numeric reasoning tasks. Our evaluation of ten model configurations from OpenAI, Google, and Anthropic on financial spreadsheets, including complex layouts, fund dividers, and multi-line column names, reveals that no standalone model achieves error rates low enough for unsupervised use in professional finance applications. The best-performing model, Gemini 3.1 Pro, achieves 82.4% accuracy across twenty-four evaluation files of varying complexity and structural layout (approximately 1 error per 6 questions), followed by GPT-5.2 with reasoning at 80.4%, Claude Opus 4.6 with thinking at 80.2%, and Gemini 3 Pro at 80.2%. Performance degrades substantially on larger, more complex spreadsheets: the largest spreadsheet (152 companies, 8 funds) yields an average accuracy of just 48.6% across all models, compared to 86.2% on the easiest evaluation file. These difficulty patterns are consistent across all ten models, indicating that they reflect LLM limitations rather than idiosyncratic model weaknesses. Reliable financial spreadsheet extraction will likely require architectural approaches that separate document understanding from deterministic computation. 1 Introduction Large Language Models (LLMs) such as GPT-5 ( OpenAI 2025 ) , Gemini 3.1 Pro ( Google 2026 ) , and Claude Opus 4.6 ( Anthropic 2026 ) now routinely match or exceed human performance on many text-based tasks. In the financial services industry, practitioners already use LLMs in document-heavy workflows: analyzing legal agreements, extracting key clauses from private placement memoranda, summarizing due diligence questionnaires, and drafting investment memos ( Wu et al. 2023 ; Lee et al. 2025 ; Lopez-Lira et al. 2025 ) . 1.1 The Due Diligence Bottleneck Alternative investment due diligence remains one of the most resource-intensive processes in finance. Evaluating a single private equity or venture capital fund can require significant analyst effort and cost. 1 1 1 Industry practitioners report figures ranging from 55–120 analyst hours and $20k–$100k per fund evaluation, with total pre-investment costs of $200k–$1M per allocation. These estimates are based on practitioner experience rather than published surveys. With global private markets assets under management now exceeding $24 trillion ( McKinsey 2026 ) , Limited Partners (LPs) need tools to scale diligence without sacrificing rigor. While LLMs can significantly accelerate text-based diligence tasks (document review, clause comparison, compliance checking, and memo generation), a persistent challenge is the extraction of structured financial data from Excel spreadsheets. Portfolio monitoring spreadsheets, cash flow statements, and track record files are central to fund evaluation, yet these files have no standard format. They vary in sheet structure, column naming, header placement, data layout, and the use of formatting conventions that convey semantic meaning. 1.2 The Spreadsheet Challenge Financial spreadsheets present several challenges that distinguish them from both unstructured text and simplified tabular benchmarks: 1. Complex multi-sheet structures : Real financial models span multiple worksheets with complex cross-references, fund dividers, and section headers embedded within data rows. 2. Non-standard layouts : Unlike clean CSV datasets, financial spreadsheets contain merged cells, hierarchical multi-row headers, embedded formulas, footnotes, and visual formatting that conveys semantic meaning. 3. Domain-specific conventions : Financial terminology, calculation methodologies, and presentation standards vary across institutions and asset classes. 4. Low tolerance for errors : In professional finance, even small numerical errors can have significant consequences. A 5% error in a valuation metric could misrepresent millions in investment decisions. Industry practitioners typically expect very high accuracy (often cited as \sim 97%+) for automated financial workflows. 2 2 2 This threshold is an operational heuristic based on practitioner experience, not a formally defined regulatory standard. At 97% accuracy on a 100-question extraction, approximately 3 answers would still be wrong, requiring human review. At the 82.4% accuracy of our best model, roughly 1 in 6 questions is wrong, meaning the review burden may be approaching that of manual extraction. See Section 7 for further discussion. 1.3 The Dataset Gap A key obstacle to progress in financial spreadsheet extraction is the lack of publicly available, real-world benchmark datasets. Private equity fund data rooms are confidential by nature. LPs sign non-disclosure agreements, and fund managers guard their portfolio data closely. This confidentiality makes it impossible to create public benchmarks from real data, leaving researchers without a standardized way to evaluate extraction systems. Existing tabular benchmarks such as WikiTableQuestions ( Pasupat & Liang 2015 ) , Spider ( Yu et al. 2018 ) , and TabFact ( Chen et al. 2020 ) use simplified, well-structured tables that do not capture the complexity of real financial spreadsheets. Even financial-specific benchmarks like FinQA ( Chen et al. 2021 ) and TAT-QA ( Zhu et al. 2021 ) focus on public company filings rather than the messy, multi-sheet workbooks used in private markets. 2 Related Work 2.1 LLMs on Tabular Data Prior work has explored LLM capabilities on tabular data through benchmarks like WikiTableQuestions ( Pasupat & Liang 2015 ) , Spider ( Yu et al. 2018 ) , and TabFact ( Chen et al. 2020 ) , focusing on semantic parsing, SQL generation, and fact verification over relatively clean tabular data. More recent work has examined LLM performance on spreadsheet tasks specifically: SheetCopilot ( Li et al. 2023 ) and SpreadsheetLLM ( Dong et al. 2024 ) propose specialized architectures for spreadsheet understanding, TableBench ( Wu et al. 2025 ) reveals that even frontier LLMs fall short of human performance across 18 fields of table QA, and the SemEval-2025 shared task ( Osés Grijalba et al. 2025 ) provides a recent community-wide evaluation of LLM tabular reasoning. Closest to our setting, TableLLM ( Zhang et al. 2025 ) evaluates tabular data manipulation in real office scenarios, though on well-structured tables rather than the complex, multi-fund financial layouts we target. These benchmarks share a common limitation: they use simplified, well-structured data that does not capture the complexity of real-world financial models. We focus specifically on private equity portfolio analysis, where spreadsheet structures are irregular and error tolerance is minimal. 2.2 Financial NLP and Document Understanding Financial NLP spans sentiment analysis ( Araci 2019 ) , document classification ( Loukas et al. 2022 ) , and information extraction from financial reports ( Chen et al. 2021 ) . FinQA ( Chen et al. 2021 ) and TAT-QA ( Zhu et al. 2021 ) specifically address question answering over financial documents with tables, but focus primarily on public company filings rather than the complex spreadsheet models used in private markets. Li et al. ( Li et al. 2025 ) introduce a framework for automated financial data extraction from unstructured documents using LLMs, while Ke et al. ( Ke et al. 2025 ) survey the growing field of LLMs applied to document intelligence more broadly. Recent

Read the complete text