SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension

Junjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou, Mo Yu · 2025

Read SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension by Junjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou, Mo Yu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01959.txt.

Introduction

Wu and co-authors present a computer method, published in 2025, for searching long documents with the help of context. Search systems that feed text to chatbots usually chop documents into small chunks. Chunks lose the thread of the whole document, so search can misread them. One fix is to store each chunk with a longer stretch of context baked in. That fix strains the model and still returns blunt results. The team's method aims to do better.

Their system builds chunk summaries that keep the sense of the surrounding story. Search then matches questions to chunks that carry their context with them. The authors test the method on tasks that need grasp of long stories and loose links between ideas. It beats plain chunk search on those tasks. The work matters for tools that must answer from books, records, or long reports.

Catalog record

CREATORJunjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou, Mo Yu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-01959

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehension Junjie Wu, Jiangnan Li, Yuqing Li, Lemao Liu, Liyan Xu, Jiwei Li, Dit-Yan Yeung, Jie Zhou, Mo Yu Abstract Retrieval-augmented generation (RAG) over long documents typically involves splitting the text into smaller chunks, which serve as the basic units for retrieval. However, due to dependencies across the original document, contextual information is often essential for accurately interpreting each chunk. To address this, prior work has explored encoding longer context windows to produce embeddings for longer chunks. Despite these efforts, gains in retrieval and downstream tasks remain limited. This is because (1) longer chunks strain the capacity of embedding models due to the increased amount of information they must encode, and (2) many real-world applications still require returning localized evidence due to constraints on model or human bandwidth. We propose an alternative approach to this challenge by representing short chunks in a way that is conditioned on a broader context window to enhance retrieval performance – i.e. , situating a chunk’s meaning within its context. We further show that existing embedding models are not well-equipped to encode such situated context effectively, and thus introduce a new training paradigm and develop the situated embedding models ( SitEmb ) . To evaluate our method, we curate a book-plot retrieval dataset specifically designed to assess situated retrieval capabilities. On this benchmark, our SitEmb-v1 model based on BGE-M3 substantially outperforms state-of-the-art embedding models, including several with up to 7-8B parameters, with only 1B parameters. Our 8B SitEmb-v1.5 model further improves performance by over 10% and shows strong results across different languages and several downstream applications. 1 1 1 https://huggingface.co/SituatedEmbedding 1 Introduction Text embedding models ( Wang et al. 2024a ; Sturua et al. 2024 ; Nussbaum et al. 2024 ; Hu et al. 2025 ) encode textual inputs into vector spaces. These models enable efficient semantic representation and matching, thus are foundational to many applications involving retrieval-augmented generation (RAG) ( Lewis et al. 2020 ) , such as code generation ( Wang et al. 2024b ; Miao et al. 2024 ) , reference generation ( Wu et al. 2025 ) , and personal AI assistants ( Martin & Johnson 2023 ) . In these tasks, candidate documents are typically segmented into smaller chunks to facilitate efficient processing. However, since documents often exhibit a narrative or logical flow, the meaning of each chunk is highly dependent on its surrounding context. This highlights the need for text embeddings that capture broader contextual information to enable context-aware retrieval . One straightforward approach to this issue is to increase chunk size, allowing each chunk to capture more information. This has motivated a wave of recent work on supporting long input sequences in embedding models, either by designing efficient bidirectional models ( Chen et al. 2024 ; Sturua et al. 2024 ; Nussbaum et al. 2024 ) , or by repurposing powerful unidirectional pre-trained LLMs as embedding generators ( Li et al. 2023 ; Wang et al. 2024a ; Moreira et al. 2024 ; Kim et al. 2024 ) . These models can produce embeddings for sequences of up to 8,192 tokens or more. Figure 1: Comparison of the same embedding models that return the same lengths of texts with different chunk sizes on our evaluation task (§ 3 ). X-axis refers to chunk sizes. For example, when the return text is 5k and the chunk size is 1,024, the retriever returns top-5 chunks. However, it is often observed that simply enabling longer input windows does not necessarily lead to better embeddings . A key reason lies in the limited capacity of embedding vectors – embedding models must compress the information in the input text into a single vector. Intuitively, the longer the input chunk, the more information it contains, and the more long-range dependencies across arbitrary pairs of chunks within a document it needs to capture. This increases the likelihood of critical information loss during compression. Existing models are trained by merely extending the context window, without explicitly learning how to represent such distributed contextual relationships, which leads to a counterintuitive outcome: applications built on long-chunk embeddings often underperform those using short-chunk embeddings, despite the latter discarding more contextual information. Figure 1 illustrates this effect on a book plot retrieval task ( Xu et al. 2024a ) . When the same total length of text (5k/10k/25k tokens) is retrieved using the same embedding model (Jina or NV-Embed), recall consistently decreases as the documents are segmented into longer chunks. Given the aforementioned challenges, we propose an alternative approach to context-aware embedding: directly incorporating the broader context surrounding each short chunk into its chunk embedding. This allows the model to account for how the chunk is situated within the original document, enabling more contextually informed embeddings. In other words, we aim to situate a chunk’s meaning within its broader context ( Yu et al. 2023 ; Xu et al. 2024a ) during the embedding process. We refer to this approach as situated embedding . By doing so, we alleviate the issue of capacity limitations: during encoding, the model only needs to identify and integrate context that is relevant to the target chunk, which is a more tractable task than modeling all dependencies across an extended input window. Building on the idea of situated embedding, we first investigate whether existing embedding models can effectively generate situated embeddings. However, we find that situated embedding cannot be achieved simply by prompting existing embedding models , as demonstrated in § 4 . To address this limitation, we develop a dedicated situated embedding model specifically designed to handle this scenario. We achieve this through two techniques: (1) Constructing context-dependent training instances using publicly available user-annotated book notes. Platforms such as Douban 2 2 2 https://book.douban.com/ allow users to write notes anchored to particular book segments. We treat the note as a query and the anchor text as groundtruth, framing a retrieval task with ~1.6M query-candidate pairs. As user notes typically reflect the contextual understanding of surrounding context, it makes context-aware embeddings beneficial for this retrieval task. (2) Promoting context usage through residual learning . In many cases, a chunk alone may offer partial (usually ambiguous) clues about its relevance to the query, allowing models to exploit shortcuts. To counter this, we employ a residual architecture where the situated embedding model is trained to resolve the residual from a baseline chunk-only embedding model. This encourages the model to focus on the additional contextual information. To evaluate models’ context-aware embedding capability, we curate a book-plot retrieval task following ( Xu et al. 2024a ) , which has been verified by previous work for its guaranteed requirement of context-aware embedding capability. Experiments demonstrate our situated embedding model’s superior performance over all the state-of-the-art embedding models, including those with up to 7B parameters and with massive pre-training. We further illustrate the generalizability of the trained models on the Recap Snippet Identification task ( Li et al. 2024 ) , a distinct task in book understanding beyond standard query-candidate retrieval, and several downstreaming story understanding tasks such as QA and claim verification that require RAG. 2 Our Situated Embedding Model We develop the first model that generates high-quality situated

Read the complete text