EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements
Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha · 2025
Read EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements by Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-08762.txt.
Introduction
Issa Sugiura and co-authors build a finance test for language models from Japanese company filings, in this 2025 paper. Models ace math and coding tests, but finance lacks hard public benchmarks. Their EDINET-Bench draws on ten years of annual reports that firms file in Japan's disclosure system.
Tasks include spotting accounting fraud, forecasting earnings, and classifying industries. Each task forces the model to read whole reports and join facts across tables and text. Even top models barely beat a simple statistical baseline on the yes-no tasks. The authors argue that raw report access is not enough. They call for test setups that mirror how finance professionals actually work.
Catalog record
| CREATOR | Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-08762
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements Issa Sugiura, Takashi Ishida, Taro Makino, Chieko Tazuke, Takanori Nakagawa, Kosuke Nakago, David Ha Abstract Financial analysis presents complex challenges that could leverage large language model (LLM) capabilities. However, the scarcity of challenging financial datasets, particularly for Japanese financial data, impedes academic innovation in financial analytics. As LLMs advance, this lack of accessible research resources increasingly hinders their development and evaluation in this specialized domain. To address this gap, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate the performance of LLMs on challenging financial tasks including accounting fraud detection, earnings forecasting, and industry prediction. EDINET-Bench is constructed by downloading annual reports from the past 10 years from Japan’s Electronic Disclosure for Investors’ NETwork (EDINET) and automatically assigning labels corresponding to each evaluation task. Our experiments reveal that even state-of-the-art LLMs struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight significant challenges in applying LLMs to real-world financial applications and underscore the need for domain-specific adaptation. Our dataset, benchmark construction code, and evaluation code is publicly available to facilitate future research in finance with LLMs. https://huggingface.co/datasets/SakanaAI/EDINET-Bench https://github.com/SakanaAI/edinet2dataset https://github.com/SakanaAI/EDINET-Bench Figure 1 : Overview of EDINET-Bench. 1 Introduction Large Language Models (LLMs) have recently demonstrated remarkable capabilities across various domains, including general knowledge, mathematics, and coding ( OpenAI 2024 ; Anthropic 2024 ; Google 2024 ; DeepSeek-AI 2025 ) . LLMs are also being explored for applications in the financial sector, where they have shown strong performance in various benchmark tasks ( Wu et al. 2023 ; Yang et al. 2023 ; Xie et al. 2023 ) . However, most existing benchmarks focus on relatively simple tasks such as information extraction and question answering ( Chen et al. 2021 ; Chen et al. 2022b ; Islam et al. 2023 ) . To make a significant societal impact in the financial domain, LLMs must excel in expert-level decision-making tasks such as accounting fraud detection and earnings forecasting ( Beneish 1999 ; Song et al. 2016 ; Kondo et al. 2019 ; Penman and Sougiannis 1998 ; Monahan 2018 ) . In addition, current financial benchmarks predominantly focus on English-speaking markets (US and Europe) or Chinese-language content. Japan, the fourth largest economy globally by nominal GDP (in 2023; World Bank 2025 ), plays a crucial role in the global economy, yet remains underserved and underrepresented by AI technology developments. Japanese financial benchmarks could significantly advance digital transformation efforts throughout the country. To address these gaps, we introduce EDINET-Bench, an open-source Japanese financial benchmark designed to evaluate LLMs on complex financial tasks. We construct EDINET-Bench using Japan’s Electronic Disclosure for Investors’ NETwork (EDINET) 1 1 1 https://disclosure2.edinet-fsa.go.jp/ , an electronic system that digitizes and publicly discloses securities reports to improve market efficiency and information accessibility. EDINET-Bench features challenging tasks including accounting fraud detection and earnings forecasting, which require sophisticated financial understanding. Our benchmark is distinctive for its automated construction and exclusive Japanese language content. As the dataset construction is automatic, we can easily incorporate future annual reports. Using EDINET-Bench, we evaluate the performance of frontier LLMs in a zero-shot setting. Our results indicate that even state-of-the-art models struggle, performing only slightly better than logistic regression in binary classification for fraud detection and earnings forecasting. These results highlight current LLM limitations in complex financial analysis and underscore the need for dedicated research to address these challenges. By open-sourcing our benchmark dataset and toolkit, we enable researchers to tackle challenging financial tasks using AI and facilitate continuous development of the benchmark. 2 Background and related work Financial benchmarks Many financial benchmarks have been proposed in order to evaluate the financial knowledge and reasoning abilities of LLMs ( Chen et al. 2021 ; Islam et al. 2023 ; Xie et al. 2024 ; Guo et al. 2025 ) . For example, FinQA ( Chen et al. 2021 ) and ConvFinQA ( Chen et al. 2022b ) are closed numerical reasoning question answering tasks related to financial analysis. Similarly, FinanceBench ( Islam et al. 2023 ) is an open-book financial question answering benchmark dataset, and FinBen ( Xie et al. 2024 ) is a benchmark-suit that contains 24 tasks. FAMMA ( Xue et al. 2025 ) uses university textbooks and the CFA exam to construct a multimodal question answering benchmark. Several benchmarks exist in the Japanese financial domain (see Nakagawa et al. 2025 for a recent survey), including Japanese-lm-fin-harness ( Hirano 2024 ) and JCPA dataset ( Masuda et al. 2023 ) , which focus on Japanese financial exam questions. Most of these datasets rely on financial experts to design high-quality questions, and therefore cannot be easily extended to include up-to-date questions after the publication. On the other hand, our framework can easily incorporate future financial documents, and automatically annotates the labels for each task. Accounting fraud detection Accounting fraud detection involves identifying deliberate misstatements, manipulations, or discrepancies in financial reports. These reports are critical for numerous stakeholders, as investors base allocation decisions on them, while job seekers evaluate potential employers through their financial disclosures ( Sou 2018 ) . Despite preliminary audits, fraudulent activities are frequently discovered only after publication, necessitating subsequent corrections ( Japan Institute of Certified Public Accountants 2024 ) . Accounting fraud detection has been studied for many years, with pioneering work by Beneish 1999 . After that, traditional machine learning approaches have been applied to accounting fraud detection ( Perols 2011 ; Dechow et al. 2011 ; Song et al. 2016 ; West and Bhattacharya 2016 ; Kondo et al. 2019 ) , but effectively processing textual information alongside numerical data in annual reports using LLMs remains underexplored. Earnings forecasting While earnings forecasting typically involves predicting the numeric magnitude of future earnings ( Penman and Sougiannis 1998 ; Monahan 2018 ) , a large body of work instead focuses on predicting the sign of the change in earnings ( Ou and Penman 1989 ; Chen et al. 2022a ) , which remains a difficult task for professionals. Kim et al. 2024 investigated if GPT-4, when provided with financial statements in which company names and fiscal years are anonymized, and without using any textual information, will outperform human analysts in predicting the direction of earnings for the following year. However, it is based on proprietary data. On the other hand, we will publish both the evaluation dataset and evaluation code to facilitate future research in this area. Industry prediction Industry prediction is a multi-class classification task that aims to predict the industry category based on security reports. While listed firms already are tagged with industry labels, e.g., by Securities Identification Code Committee (SICC), portfolio managers may want to adopt their own industry definitions, e.g., see Kimura and Nakagawa 2022 for a data-driven approach, or anticipate reclassifications as firms’ business evolve or mergers reshape their operations. Evaluating whether LLMs can predict industries based on financial information such as balance sheets and profit/loss statements serves as an effective task for measuring LLMs’ financial domain knowledge. Van Der Heijden 2022 ; Dolphin et al. 2023 used machine learning for this application. Table 1: Sample of TSV records. Some columns are omitted for ease of