FinTagging: An LLM-ready Benchmark for Extracting and Structuring Financial Information
Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, Qianqian Xie · 2025
Read FinTagging: An LLM-ready Benchmark for Extracting and Structuring Financial Information by Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, Qianqian Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-20650.txt.
Introduction
Yan Wang and co-authors build a test set for AI tools that tag numbers in company reports in a 2025 paper. Reports must tag each figure with a concept from a huge shared list, a slow and error-prone job. Old test sets shrank the task to a small slice of concepts.
Their benchmark, FinTagging, keeps the full list of over ten thousand United States concepts. It splits the job in two: first find each figure and its type in text and tables, then link it to the right concept. Tests of many language models in a no-example mode showed a split result. Models found figures well but stumbled on the fine linking step. The set gives a fairer yard stick for finance AI.
Catalog record
| CREATOR | Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, Qianqian Xie |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-20650
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FinTagging: An LLM-ready Benchmark for Extracting and Structuring Financial Information Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, Qianqian Xie Abstract We introduce FinTagging , the first full-scope, table-aware XBRL benchmark designed to evaluate the structured information extraction and semantic alignment capabilities of large language models (LLMs) in the context of XBRL-based financial reporting. Unlike prior benchmarks that oversimplify XBRL tagging as flat multi-class classification and focus solely on narrative text, FinTagging decomposes the XBRL tagging problem into two subtasks: FinNI for financial entity extraction and FinCL for taxonomy-driven concept alignment. It requires models to jointly extract facts and align them with the full 10k+ US-GAAP taxonomy across both unstructured text and structured tables, enabling realistic, fine-grained evaluation. We assess a diverse set of LLMs under zero-shot settings, systematically analyzing their performance on both subtasks and overall tagging accuracy. Our results reveal that, while LLMs demonstrate strong generalization in information extraction, they struggle with fine-grained concept alignment, particularly in disambiguating closely related taxonomy entries. These findings highlight the limitations of existing LLMs in fully automating XBRL tagging and underscore the need for improved semantic reasoning and schema-aware modeling to meet the demands of accurate financial disclosure. Code is available at our GitHub repository 1 1 1 https://github.com/The-FinAI/FinTagging and data is at our Hugging Face repository 2 2 2 https://huggingface.co/collections/TheFinAI/fintagging-68270132372c6608ac069bef . 1 Introduction Automated tagging is essential in financial reporting, converting structured content such as tables and text into machine-readable data by linking each number to its meaning. Globally, over 2 million companies publish financial reports disclosing earnings, expenses, and liabilities, which provide essential information for investors, regulators, and analysts. However, inconsistent terminology poses challenges for reliable interpretation. To address this, the eXtensible Business Reporting Language (XBRL) was introduced in 1999 as a global standard for machine-readable financial data richards2006introduction . By applying structured tags, XBRL transforms fragmented disclosures into a consistent format, enabling seamless data exchange, cross-entity comparisons, and large-scale analysis. Yet, as shown in Figure 1 , accurately tagging over 2,000 numerical facts per report to more than 10,000 standardized concepts remains labor-intensive and error-prone. In 2023 alone, over 6,500 tagging errors were identified across 33,000 SEC filings 3 3 3 https://xbrl.us/wp-content/uploads/2023/03/DQC-SECMeetingNotes-20240314.pdf . Figure 1 : An example of Financial Tagging. Existing approaches to automated XBRL tagging rely on task-specific models and pretrained language models (PLMs) loukas2022finer ; sharma2023financial , but they require costly annotations, struggle to generalize to new filings, and perform poorly as taxonomies scale. In contrast, recent large language models (LLMs) show strong zero- and few-shot reasoning without task-specific supervision wei2022chain ; cao2025development ; wang2025ordrankbennovelrankingbenchmark ; peng2025plutusbenchmarkinglargelanguage , yet their application to XBRL tagging remains underexplored due to two major limitations (Table 1 ). First, task framing is oversimplified : prior benchmarks like FiNER loukas2022finer and FNXL sharma2023financial treat tagging as extreme multi-class classification over flat label sets, offering little context for disambiguating fine-grained financial facts. These datasets typically cover only 1k+ concepts, leaving most of the 10k+ US-GAAP taxonomy untested. As shown in Table 8 , SOTA LLMs such as DeepSeek-V3 liu2024deepseek and GPT-4o hurst2024gpt achieve 0.0 precision, recall, and F1 under this setting. Second, structured data is ignored : existing datasets exclude tables, despite their central role in financial reporting. Many key facts appear in structured formats, as shown in recent financial QA benchmarks chen2021finqa ; zhu2021tat ; mathew2021docvqa , making this omission a key gap between benchmarks and real-world tagging needs. Table 1 : Detailed comparison of financial NLP benchmarks across task types, sources, and structural capabilities. “QA” means the question-answering task. “Num. Reasoning” indicates the numerical reasoning. “Struct. IE” denotes the structured information extraction. Benchmark Scenario Data Source Task Modality #Entity Label #Taxonomy Label Num. Reasoning? Struct. IE? Concept Linking? FinQA chen2021finqa Decision making SEC 10-K QA text/table 0 0 ✓ ✗ ✗ ConvFinQA chen2022convfinqa Decision making SEC 10-K QA text/table 0 0 ✓ ✗ ✗ TAT-QA zhu2021tat Financial analysis Chinese financial reports QA text/table 0 0 ✓ ✗ ✗ DocVQA mathew2021docvqa Enterprise automation Financial document images QA text/image 0 0 ✓ ✗ ✗ FiNER-ORD shah2023finer Financial tagging Chinese financial disclosures Classification text 49 0 ✗ ✗ ✗ FinRED sharma2022finred Knowledge graph construction English financial news Classification text 38 0 ✗ ✗ ✗ FiNER loukas2022finer XBRL tagging SEC 10-K Extreme classification text 0 139 ✓ ✗ ✗ FNXL sharma2023financial XBRL tagging SEC 10-K Extreme classification text 0 2,800 ✓ ✗ ✗ FinTagging (ours) Financial & XBRL tagging SEC 10-K IE + Alignment text/table 5 17,388 ✓ ✓ ✓ To address these issues, we present FinTagging , the first LLM-ready benchmark for full-scope, structure-aware XBRL tagging, which challenges models to perform end-to-end fact extraction and taxonomy alignment on corporate filings. As shown in Table 1 , unlike prior work shah2023finer ; loukas2022finer ; sharma2023financial that frames tagging as flat multi-class classification and focuses solely on narrative text, FinTagging requires models to jointly extract financial facts and align them with the US-GAAP taxonomy across both unstructured text and structured tables. This is the first LLM-oriented formulation that scales to the full set of 10k+ taxonomy labels while avoiding the intractability of single-step classification, enabling more fine-grained evaluation of model performance. In collaboration with financial reporting experts, we decompose the task into two components: the first, structured numerical information extraction , assesses a model’s ability to identify key financial facts from both text and tables; the second, fine-grained concept linking , evaluates its ability to map each fact to the correct taxonomy concept among all semantically similar candidates. Unlike prior benchmarks limited to 1,000 frequent concepts, our setting covers the entire taxonomy, exposing the limitations of frequency-biased classification and demanding precise semantic reasoning bovee2005financial . We construct two evaluation sets: FinNI-eval 4 4 4 https://huggingface.co/datasets/TheFinAI/FinNI-eval for numerical fact extraction and FinCL-eval 5 5 5 https://huggingface.co/datasets/TheFinAI/FinCL-eval for concept linking, both derived from real XBRL submissions and annotated with gold-standard mappings. We also introduce a unified evaluation framework that jointly measures extraction accuracy and concept alignment, providing a rigorous zero-shot assessment of LLMs’ capabilities for practical, table-aware XBRL tagging. We evaluate ten state-of-the-art LLMs under a zero-shot setting on three fronts: (1) end-to-end macro-F1 over the full FinTagging benchmark, (2) subtask-specific performance on FinNI and FinCL, and (3) ablation of our unified extraction-and-alignment evaluation framework. DeepSeek-V3 liu2024deepseek and GPT-4o hurst2024gpt achieve the highest macro-F1 scores, indicating that our benchmark design enables strong handling of both frequent and rare financial tags, an improvement over traditional PLM baselines evaluated under token-classification settings. In subtask analyses, these LLMs excel at numerical information extraction (FinNI) but continue to struggle with precise concept linking (FinCL), underscoring the remaining challenge of fine-grained semantic alignment. Crucially, our ablation study shows that, without the joint evaluation framework, even top LLMs produce invalid tagging outputs, highlighting the framework’s essential role in realistic assessment of XBRL tagging readiness. We conclude our main contributions as follows: (1) We introduce FinTagging , the first LLM-ready benchmark for full-scope, table-aware XBRL tagging with a two-stage pipeline—so that modern LLMs can be probed in true zero-shot setting. For each stage we release a new, professionally annotated evaluation set: FinNI-eval and FinCL-eval. (2) We conduct an extensive evaluation of state-of-the-art LLMs in