Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Analysis
Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann, Nicolas Flores-Herr · 2025
Read Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Analysis by Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann, Nicolas Flores-Herr (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-21700.txt.
Introduction
Sinchana Ramakanth Bhat and co-authors study chunk size in document retrieval in this 2025 paper. Retrieval systems split long documents into chunks before search. Chunk size can make or break how well the right passage is found.
Their tests span short and long datasets and several embedding models. Small chunks worked best for short fact-based answers. Large chunks helped when a question needs broad context to answer. Models differed in taste: one gained from large chunks and global context, while another did better with small chunks and fine matches. The authors stress that chunk size, model, and data type must be chosen together.
Catalog record
| CREATOR | Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann, Nicolas Flores-Herr |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-21700
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Rethinking Chunk Size for Long-Document Retrieval: A Multi-Dataset Analysis Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann, Nicolas Flores-Herr Abstract Chunking is a crucial preprocessing step in retrieval-augmented generation (RAG) systems, significantly impacting retrieval effectiveness across diverse datasets. In this study, we systematically evaluate fixed-size chunking strategies and their influence on retrieval performance using multiple embedding models. Our experiments, conducted on both short-form and long-form datasets, reveal that chunk size plays a critical role in retrieval effectiveness - smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers, whereas larger chunks (512-1024 tokens) improve retrieval in datasets requiring broader contextual understanding. We also analyze the impact of chunking on different embedding models, finding that they exhibit distinct chunking sensitivities. While models like Stella benefit from larger chunks, leveraging global context for long-range retrieval, Snowflake performs better with smaller chunks, excelling at fine-grained, entity-based matching. Our results underscore the trade-offs between chunk size, embedding models, and dataset characteristics, emphasizing the need for improved chunk quality measures, and more comprehensive datasets to advance chunk-based retrieval in long-document Information Retrieval (IR). 1 Introduction Retrieval-Augmented Generation (RAG) has emerged as a powerful paradigm in natural language processing (NLP), enabling large language models (LLMs) to enhance response accuracy by incorporating relevant external knowledge retrieved from document corpora [ 1 ] [ 2 ] . This approach has significantly improved performance in knowledge-intensive tasks by mitigating the limitations of parametric memory in LLMs and enhancing factual consistency [ 3 ] . The effectiveness of RAG systems heavily depend on document chunking strategies, which segment textual data into manageable units before retrieval. Among various chunking techniques, fixed-size token-based chunking remains a prevalent method due to its simplicity and ease of implementation [ 4 ] . Fixed-size chunking segments documents into uniform token-length chunks, ensuring compatibility with transformer-based architectures that have strict token limits [ 5 ] . However, despite its widespread adoption, the robustness of this approach across varied document lengths remains underexplored. The effectiveness of fixed-size chunking can be influenced by factors such as information dispersion across chunks, redundancy in retrieval, and retrieval precision [ 6 ] . Understanding these factors is crucial to optimizing retrieval performance and downstream generation quality in RAG applications. The importance of dataset characteristics in retrieval performance has been well established. Some datasets exhibit high answer locality, where relevant spans are concentrated within a few sentences, while others require reasoning over long contexts spanning multiple chunks [ 7 ] . Recent studies have highlighted that datasets with diverse document structures demand adaptive chunking strategies to optimize retrieval effectiveness [ 8 ] . Additionally, the role of embedding models in retrieval varies - encoder-based models tend to benefit from exact term matching, whereas decoder-based models leverage broader contextual cues [ 9 ] . Despite its widespread use, fixed-size chunking may not be optimal for all datasets. For instance, hierarchical chunking methods have been shown to improve retrieval by preserving semantic coherence within chunks [ 10 ] . Moreover, the challenge of evaluating chunking strategies stems from the difficulty of establishing ground truth relevance in retrieval [ 11 ] . While prior work has explored chunking in traditional retrieval systems, there is limited research on how fixed-size chunking impacts retrieval performance across documents of varying domains in modern RAG architectures. We address this research gap through a structured set of ablations on different domain-specific datasets across different chunk sizes and embedding models. For reproducibility of results, our code is available on Github 1 1 1 Code and interface description: https://github.com/fraunhofer-iais/chunking-strategies . Our key contributions can be summarized as follows: • We systematically evaluate the impact of fixed-size chunking on retrieval effectiveness across multiple datasets, analyzing recall@k trends for different chunk sizes and embedding models. • We investigate how document length, domain, and dataset diversity influence retrieval performance, providing insights into chunk size selection for different question-answering scenarios. • We explore retrieval biases across different embedding models, highlighting how chunking strategies interact with model architectures to optimize retrieval outcomes of chunk size. 2 Related Work Table 1: Overview of dataset statistics, including key properties. Datasets that were stitched together to create longer synthetic documents are marked with an asterisk (*) next to their names. Dataset # Docs Q/Doc Tokens/Doc Tokens/Q Tokens/A Unique Tokens/Doc NarrativeQA 1073 1.9 51830.4 8.5 12.6 8983.5 Natural Questions (NQ) 25010 1.0 6918.2 9.0 6.8 2864.8 NewsQA* 685 11.9 8484.5 6.5 5.2 3265.3 COVID-QA* 55 2.1 10009.2 8.8 11.1 2904.5 TechQA* 45 10.6 7597.2 51.1 46.9 2130.1 SQuAD* 306 43.7 7998.3 9.9 3.9 2949.3 Chunking Strategies : The effective segmentation of long documents into manageable chunks is a persistent challenge in information retrieval. We recognize the advancements in dynamic chunking [ 12 ] and neural-integrated chunking strategies [ 7 ] , however, fixed-size chunking remains a widely used and foundational approach due to its simplicity and computational efficiency. Furthermore, understanding the behavior and limitations of fixed-size chunking provides a crucial baseline for evaluating the effectiveness of more complex methods. Therefore, we conduct a comprehensive empirical analysis of fixed-size token chunking across a diverse range of modern question answering datasets. Embedding Models : Transformer-based embedding models form the backbone of modern retrieval systems, encoding text into dense representations for similarity-based retrieval. Early works leveraged pre-trained models like BERT [ 13 ] and RoBERTa [ 14 ] for retrieval, while more recent approaches introduced contrastive learning and hard negative mining to improve retrieval quality [ 15 ] [ 16 ] . Dense retrieval models such as DPR [ 15 ] and Contriever [ 17 ] have demonstrated significant improvements over sparse retrieval methods by leveraging deep contextualized representations. However, embedding models exhibit distinct retrieval biases based on their architectural design. Positional encoding mechanisms play a crucial role in how transformers interpret document structure and chunked inputs. Standard transformers use absolute or sinusoidal positional encodings [ 18 ] , while newer approaches like Rotary Positional Encoding (RoPE) [ 19 ] and Attention with Linear Biases (ALiBi) [ 20 ] modify the handling of positional information. These encodings impact retrieval behavior, especially in chunk-based retrieval settings. Dataset Considerations in Chunk-Based Retrieval : Understanding the characteristics of question answering datasets is vital for developing robust and generalizable retrieval systems. Recent studies have emphasized the importance of analyzing dataset biases and evaluating model performance across diverse question types [ 21 ] . Furthermore, researchers have explored the creation of challenging benchmark datasets that require reasoning over long contexts and multiple documents [ 22 ] . We extend this line of research by conducting a detailed analysis of dataset characteristics and examining their correlation with retrieval performance. Additionally, we investigate the impact of stitching QA pairs within small datasets to form long documents, providing insights into real-world retrieval challenges. This approach is inspired by recent work that has demonstrated the feasibility of stitching datasets for similarity-based evaluations [ 11 ] 3 Methodology Chunking and Retrieval We employ a RAG system built using LlamaIndex [ 23 ] to evaluate the impact of chunk size on retrieval performance across various datasets. Our retrieval process begins with fixed-size token splitting, where each document in each dataset is segmented into chunks of predetermined lengths using LlamaIndex’s TokenTextSplitter . This method ensures