JFinTEB: Japanese Financial Text Embedding Benchmark Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia

Masahiro Suzuki, Hiroki Sakaji · 2026

Read JFinTEB: Japanese Financial Text Embedding Benchmark Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia by Masahiro Suzuki, Hiroki Sakaji (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-15882.txt.

Introduction

Suzuki and Sakaji introduce JFinTEB, the first broad benchmark for judging Japanese finance text embeddings, numeric stand-ins for text meaning. Existing benchmarks cover little of the quirks found in Japanese finance texts. JFinTEB spans retrieval and sorting tasks drawn from realistic settings, including sentiment analysis and document sorting from economic survey data. Tests across Japanese, multilingual, and commercial models set a standard protocol, and the datasets and framework are public.

Catalog record

CREATORMasahiro Suzuki, Hiroki Sakaji
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23762
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.15882

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

JFinTEB: Japanese Financial Text Embedding Benchmark Conference: Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 20–24, 2026; Melbourne, VIC, Australia Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’26), July 20–24, 2026, Melbourne, VIC, Australia Masahiro Suzuki, Hiroki Sakaji Abstract . We introduce JFinTEB, the first comprehensive benchmark specifically designed for evaluating Japanese financial text embeddings. Existing embedding benchmarks provide limited coverage of language-specific and domain-specific aspects found in Japanese financial texts. Our benchmark encompasses diverse task categories including retrieval and classification tasks that reflect realistic and well-defined financial text processing scenarios. The retrieval tasks leverage instruction-following datasets and financial text generation queries, while classification tasks cover sentiment analysis, document categorization, and domain-specific classification challenges derived from economic survey data. We conduct extensive evaluations across a wide range of embedding models, including Japanese-specific models of various sizes, multilingual models, and commercial embedding services. We publicly release JFinTEB datasets and evaluation framework at https://github.com/retarfi/JFinTEB to facilitate future research and provide a standardized evaluation protocol for the Japanese financial text mining community. This work addresses a critical gap in Japanese financial text processing resources and establishes a foundation for advancing domain-specific embedding research. 1. Introduction Figure 1. Overview of JFinTEB benchmark. JFinTEB is a comprehensive benchmark for evaluating Japanese financial text embeddings, covering diverse tasks such as classification, retrieval, and clustering. Text embeddings provide a unified representation for textual data and underpin a wide range of text mining and natural language processing tasks. Embedding-based retrieval is a core component of modern information retrieval systems, including search, question answering, and recommendation. The development of comprehensive benchmarks has been crucial for advancing embedding quality, with established evaluations including MTEB for English ( 10 ) , MMTEB for multilingual contexts ( 3 ) , and JMTEB for Japanese ( 9 ) . Recently, domain-specific benchmarks such as FinMTEB have demonstrated the importance of specialized evaluation for financial applications in English and Chinese ( 17 ) . Despite Japan’s position as a major financial market and the growing adoption of NLP technologies in Japanese financial institutions, no unified benchmark exists for evaluating Japanese financial text embeddings. This gap represents a significant limitation for developing and deploying embedding models in Japan’s financial sector, where domain-specific language patterns, regulatory terminology, and cultural contexts require specialized evaluation. In particular, Japanese financial texts exhibit characteristics that make general-purpose benchmarks insufficient. Financial disclosures such as quarterly reports and securities filings use highly domain-specific terminology and formulaic phrasing that rarely appear in general Japanese corpora. Such linguistic phenomena often arise in scenarios where embeddings are directly used for retrieval, clustering, or zero-shot classification—settings common in financial information systems but underrepresented in existing benchmarks. These considerations underscore the necessity of a dedicated benchmark to systematically evaluate embedding models for Japanese finance. Existing benchmarks face limitations when applied to Japanese financial contexts. JMTEB ( 9 ) focuses on general Japanese tasks but does not include financial applications, while FinMTEB ( 17 ) targets the financial domain but only in English and Chinese. Furthermore, financial text processing demands specialized capabilities including regulatory document analysis, market sentiment understanding, and industry classification—requirements not adequately addressed by current evaluation frameworks. Several related efforts have proposed financial NLP tasks and evaluations. NTCIR U4 ( 8 ) addresses Japanese financial document understanding through table retrieval and extraction, and Hirano ( 5 ) studies the performance of large language models on Japanese financial tasks. NTCIR FinNum ( 1 ) investigates numerical semantics in financial texts in English. While these works provide valuable task-specific insights, they are not designed to evaluate text embeddings under a unified, multi-task benchmarking framework. To address these limitations, we introduce JFinTEB (Japanese Financial Text Embedding Benchmark), the first comprehensive benchmark for evaluating Japanese financial text embeddings. Figure 1 illustrates the overall structure of our benchmark. Our benchmark comprises 11 carefully designed tasks spanning classification, retrieval, and clustering across diverse financial contexts, from regulatory documents to market sentiment analysis. We establish rigorous quality assurance protocols and evaluate 14 representative embedding models, providing baseline results and practical insights for model selection in Japanese financial applications. As a result, JFinTEB complements existing benchmarks by introducing financial-domain tasks in Japanese, filling a critical gap for both domestic applications and cross-lingual evaluation. Table 1 summarizes their scope compared to JFinTEB, highlighting how our resource complements prior work. Unlike prior benchmarks, JFinTEB focuses on the Japanese financial domain, which combines language-specific challenges with application scenarios that are critical for industry but missing in existing resources. Rather than constructing artificially difficult tasks, JFinTEB focuses on realistic and well-defined information needs observed in Japanese financial applications, where embeddings are commonly used for retrieval, clustering, and classification. Consequently, some retrieval tasks exhibit high performance under current models, highlighting the maturity of embedding methods in realistic financial settings. To our knowledge, JFinTEB is the first benchmark that systematically evaluates Japanese financial text embeddings across multiple tasks under a unified evaluation protocol. Our design enables reproducible evaluation of text embeddings under data distributions commonly encountered in practical information retrieval systems. The contributions of this work are threefold: (1) development of the first comprehensive Japanese financial text embedding benchmark with 11 validated tasks; (2) systematic evaluation of 14 embedding models including Japanese-specialized and multilingual approaches; and (3) public release of all datasets, evaluation code, and baseline results at https://github.com/retarfi/JFinTEB , to facilitate reproducible research in Japanese financial text mining. Table 1. Comparison with prior embedding benchmarks. Abbreviations: Cls = Classification, Ret = Retrieval, Clus = Clustering, STS = Semantic Textual Similarity, RR = Reranking, PC = Pair Classification, Summ = Summarization, BM = Bitext Mining. Benchmark Lang. Domain Tasks MTEB En General Cls, Ret, STS, RR, PC, Summ, BM JMTEB Ja General Cls, Ret, STS, RR, PC FinMTEB En, Zh Finance Cls, Ret, Clus, RR, PC, Summ JFinTEB (ours) Ja Finance Cls, Ret, Clus 2. JFinTEB Benchmark 2.1. Task Design and Dataset Construction 2.1.1. Classification Tasks We include several existing datasets: chABSA for aspect-based sentiment ( 15 ) , three tasks derived from the Economy Watchers Survey (domain, sentiment, horizon) ( 14 ) , MultiFin-ja for financial news headlines ( 7 ) , and Wikinews classification ( 11 ) . “Horizon” performs binary classification of EWS comments into current-state and future-outlook categories, utilizing the inherent survey structure that collects assessments for both present and prospective economic conditions. In addition, we construct two new datasets: Industry 17 and Industry 33, where company descriptions from Japanese Wikipedia are aligned with official JPX industry categories at two granularities (17 and 33 sectors), enabling coarse- and fine-grained evaluation. Company pages of Tokyo Stock Exchange Prime Market listed firms are identified by parsing the listing information template in Japanese Wikipedia articles and extracting stock codes via automated matching with the JPX database, with no manual verification. Since JPX industry classifications assign each company to exactly one sector, no ambiguity arises in label assignment. Industry labels follow the official classification published by Japan Exchange Group (JPX) 1 1 1 https://www.jpx.co.jp/english/markets/statistics-equities/misc/01.html . Wikinews classification categorizes Wikinews articles into politics and economics domains, extracted from the broader categorical structure used in previous studies ( 11 ) . 2.1.2. Retrieval Tasks Our

Read the complete text