Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems
Sejong Kim, Hyunseo Song, Hyunwoo Seo, Hyunjun KimKAISTDaejeon, South Korea {kingsj,ssongzzz,shw4166,hyunjun1121}@kaist.ac.kr · 2025
Read Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems by Sejong Kim, Hyunseo Song, Hyunwoo Seo, Hyunjun KimKAISTDaejeon, South Korea {kingsj,ssongzzz,shw4166,hyunjun1121}@kaist.ac.kr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15191.txt.
Introduction
Kim, Song, Seo, and Kim tune how a finance question tool finds the right pages before it writes its answers, in work from twenty twenty-five. Finance files, like yearly firm reports, use trade words and tables set inside tables. How well the tool finds pages caps how well it can answer. Their chain of steps tunes the hunt before, during, and after the search.
Prep work adds detail to both the asks and the file text before the hunt starts. Tuned embeddings, plus a mixed dense and sparse search, pull in more of the right pages. Preference optimization and a pick of the best files then polish the last set. Seven finance benchmarks show large gains in how well the hunt works. Answers hit the mark more and fit the ask, thanks to the sharper hunt. The full chain is shared on GitHub for others to rerun.
Catalog record
| CREATOR | Sejong Kim, Hyunseo Song, Hyunwoo Seo, Hyunjun KimKAISTDaejeon, South Korea {kingsj,ssongzzz,shw4166,hyunjun1121}@kaist.ac.kr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15191
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Optimizing Retrieval Strategies for Financial Question Answering Documents in Retrieval-Augmented Generation Systems Sejong Kim, Hyunseo Song, Hyunwoo Seo, Hyunjun KimKAISTDaejeon, South Korea {kingsj,ssongzzz,shw4166,hyunjun1121}@kaist.ac.kr Abstract Retrieval-Augmented Generation (RAG) has emerged as a promising framework to mitigate hallucinations in Large Language Models (LLMs), yet its overall performance is dependent on the underlying retrieval system. In the finance domain, documents such as 10-K reports pose distinct challenges due to domain-specific vocabulary and multi-hierarchical tabular data. In this work, we introduce an efficient, end-to-end RAG pipeline that enhances retrieval for financial documents through a three-phase approach: pre-retrieval , retrieval , and post-retrieval . In the pre-retrieval phase, various query and corpus preprocessing techniques are employed to enrich input data. During the retrieval phase, we fine-tuned state-of-the-art (SOTA) embedding models with domain-specific knowledge and implemented a hybrid retrieval strategy that combines dense and sparse representations. Finally, the post-retrieval phase leverages Direct Preference Optimization (DPO) training and document selection methods to further refine the results. Evaluations on seven financial question answering datasets—FinDER, FinQABench, FinanceBench, TATQA, FinQA, ConvFinQA, and MultiHiertt—demonstrate substantial improvements in retrieval performance, leading to more accurate and contextually appropriate generation. These findings highlight the critical role of tailored retrieval techniques in advancing the effectiveness of RAG systems for financial applications. A fully replicable pipeline is available on GitHub: https://github.com/seohyunwoo-0407/GAR . 1 Introduction Ensuring accuracy and reliability is paramount in the financial domain. Even minor errors or misinterpretations within financial statements or regulatory filings can trigger significant economic losses and adversely impact investment decisions and compliance processes. ( Rezaee 2005 ) Motivated by these high-stakes challenges, this paper presents a specialized framework that integrates Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) tailored to the unique challenges of financial data. While LLMs have demonstrated impressive performance in natural language processing, they are not immune to limitations—particularly in specialized domains where hallucinations and misinterpretations can have critical issues. ( Kerner 2024 ) . RAG is designed to mitigate these shortcomings by incorporating an additional retrieval step into the generation process. Rather than relying solely on pre-trained knowledge, the system first retrieves pertinent information from external sources and subsequently integrates this data during the generation phase. This two-stage approach ensures that the LLM has access to the most relevant information before generating its response. By leveraging external knowledge sources, RAG significantly enhances performance on domain-specific tasks. In this paper, we propose a novel RAG pipeline optimized for usage in the finance domain. Our key contributions span robust data preparation, domain adaptation, and effective information retrieval. First, we compared tailored preprocessing methods for preparing both user inputs and financial documents, ensuring our system effectively captures and utilizes context from complex and diverse data sources. Next, we utilized task-specific retrieval methods that leverages SOTA models to the unique language and structural features of financial information, thus improving retrieval performance. Finally, we implemented a reranking method , coupled with a novel method ( document selection ), to guarantee that the final generated responses are grounded in the most accurate and relevant data. 2 Related Work 2.1 Embedder Fine-tuning Fine-tuning, the process of adapting a pre-trained model to domain-specific tasks using typically smaller datasets, has been widely explored across various applications. While embedding models exhibit strong zero-shot performance on general benchmarks such as MTEB ( Muennighoff et al. 2023 ; Zhang et al. 2017 ) , recent studies have demonstrated that even modestly sized models can benefit substantially from fine-tuning when applied to domain-specific tasks. For instance, fine-tuning embedders on specialized datasets has led to notable improvements in areas such as medical question answering Sengupta et al. 2024 and financial question answering Anderson et al. 2024 . In the finance domain, prior research on embedders has underscored several inherent challenges: domain-specific vocabulary and semantic patterns, the complexity of multi-hop queries, and multimodal data (e.g. text, tables, and time-series) Tang & Yang 2024 ; Kim et al. 2024 ; Xie et al. 2024 . These challenges necessitate tailored fine-tuning strategies that can effectively capture the nuanced information contained in financial documents. Within the framework of Retrieval-Augmented Generation (RAG), embedding models are primarily tasked with Information Retrieval (IR), where the semantic similarity between a query and a corpus is assessed and ranked. A prevalent strategy for enhancing this process is contrastive learning or contrastive fine-tuning—which relies on constructing triplets (query, relevant corpus, irrelevant corpus) to form positive and negative training pairs Karpukhin et al. 2020 . Despite the effectiveness of contrastive learning in embedders Lu et al. 2024 , there remains a notable gap in the literature regarding the impact of embedder fine-tuning on RAG systems, particularly within the finance domain Setty et al. 2024 . By addressing this gap, our work aims to explore and quantify the benefits of embedder fine-tuning in RAG applications, thereby contributing to the broader understanding of domain-adaptive IR. 2.2 Pre-Retrieval Effective dataset preprocessing is essential for enhancing the performance of Retrieval-Augmented Generation (RAG) systems, because it directly impacts the clarity and semantic alignment of both queries and documents ( Gao et al. 2024 ) . Previous work has shown that short, context-poor queries can lead to significant ambiguity, which in turn hinders retrieval accuracy ( Koo et al. 2024 ) . To mitigate this issue, researchers have explored various query enhancement techniques—such as query expansion and rephrasing—to enrich the original input and better capture user intent ( Patel 2024 ) . In line with these findings, we evaluate query preprocessing methods —from raw queries and keyword extraction with linguistic simplification to LLM-based query expansion—and show that adding contextual information significantly improves retrieval performance. In addition to query enhancement, the heterogeneity of corpus data presents unique challenges that necessitate tailored preprocessing strategies. Unlike approaches that apply a uniform treatment to all documents, recent studies have emphasized the importance of adapting preprocessing to the structural characteristics of the corpus—especially when dealing with diverse formats such as plain text and tabular data. Building on these insights, we explore various corpus preprocessing methods —preserving original formats markdown restructuring, table annotation, and table extraction. Our evaluation shows that markdown restructuring yields the best performance, highlighting the benefits of an optimized, data-driven pipeline. Overall, our framework addresses query ambiguity and document heterogeneity, improving retrieval accuracy in domain-specific applications. 2.3 Hybrid retrieval Recent advancements in Retrieval-Augmented Generation (RAG) systems have increasingly focused on overcoming the limitations of using a single retrieval modality by fusing the strengths of both dense and sparse retrieval approaches. Dense retrieval methods, which leverage semantic embeddings generated by models such as BERT ( Devlin et al. 2019 ) or SentenceTransformers ( Reimers & Gurevych 2019 ) , excel at capturing deep contextual relationships between queries and documents. However, they may sometimes fail to retrieve documents that contain precise terms, proper nouns, or abbreviations. In contrast, sparse retrieval techniques, employing methods like BM25 ( Wang et al. 2021 ) , offer excellent keyword matching capabilities and provide high interpretability, although they often lack the ability to grasp nuanced semantic meaning. ( Sengupta et al. 2024 ) To address these complementary weaknesses, hybrid retrieval methods have been proposed. These methods combine the scores obtained from dense and sparse retrieval, typically through linear weighted fusion or techniques such as Reciprocal Rank Fusion (RRF).