Dynamic Injection of Entity Knowledge into Dense Retrievers

Ikuya Yamada Ryokan Ri Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, [2.5mm] Studio Ousia · 2025

Read Dynamic Injection of Entity Knowledge into Dense Retrievers by Ikuya Yamada Ryokan Ri Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, [2.5mm] Studio Ousia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-03922.txt.

Introduction

Ikuya Yamada and four co-authors upgrade search engines with outside knowledge in a 2025 paper. Dense retrievers turn queries and passages into number codes and match them, but they stumble on rare entities they know little about. Retraining the whole model for each new fact is too costly.

Their system, the Knowledgeable Passage Retriever, adds a layer that pays attention to entities in the text. Entity codes sit outside the model and can be updated on the fly, with no retraining. Tests on three data sets showed steady gains, largest on questions built around entities. Built on a popular off-the-shelf retriever, it set the best scores among models of its size on two data sets.

Catalog record

CREATORIkuya Yamada Ryokan Ri Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, [2.5mm] Studio Ousia
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-03922

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Dynamic Injection of Entity Knowledge into Dense Retrievers Ikuya Yamada Ryokan Ri Takeshi Kojima Yusuke Iwasawa Yutaka Matsuo, [2.5mm] Studio Ousia Abstract Dense retrievers often struggle with queries involving less-frequent entities due to their limited entity knowledge. We propose the Knowledgeable Passage Retriever (KPR), a BERT-based retriever enhanced with a context-entity attention layer and dynamically updatable entity embeddings. This design enables KPR to incorporate external entity knowledge without retraining. Experiments on three datasets show that KPR consistently improves retrieval accuracy, achieving a substantial 12.6% gain on the EntityQuestions dataset over the model without KPR extensions. When built on the off-the-shelf bge-base retriever, KPR achieves state-of-the-art performance among similarly sized models on two datasets. Code and models will be released soon. 1 Introduction Language models (LMs) struggle to capture less-frequent or up-to-date entity knowledge Kandpal et al. 2023 ; Mallen et al. 2023 , often resulting in hallucinations Shuster et al. 2021 . Retrieval-augmented generation (RAG), which enhances LMs by leveraging external knowledge retrieval, is a promising approach to mitigate this issue. Dense retrievers are commonly employed for this purpose; however, because they also rely on LMs, they likewise struggle with queries involving less-frequent entities Sciavolino et al. 2021 and often fail to retrieve such knowledge effectively. In this paper, we address this problem by proposing a simple extension to dense retrievers that dynamically injects entity knowledge into their embeddings. Specifically, we introduce the K nowledgeable P assage R etriever (KPR), a BERT-based dense retriever Karpukhin et al. 2020 that integrates entity embeddings adaptable at inference time (see Figure 1 ). KPR is trained to attend to entity knowledge based on the context of the input text, through a context-entity attention layer placed on top of BERT. Entity embeddings are obtained via single-pass BERT inference on texts referring to the entity and are kept frozen during training. KPR detects entities in the input text using a simple dictionary-based entity linker. Since both the entity embeddings and the linker can be modified after training, new entity knowledge can be easily added or updated without retraining. Figure 1: Architecture of KPR, a dense retriever with a context-entity attention layer that incorporates entity knowledge, given a query as input. Entity embeddings are obtained via BERT inference, and both the embeddings and the entity linker can be updated without retraining. Relevance is scored by the dot product of query and passage embeddings, both produced by KPR. To evaluate KPR, we use the EntityQuestions (EQ) dataset Sciavolino et al. 2021 , which includes many queries with less-frequent entities, as well as the Natural Questions (NQ) Kwiatkowski et al. 2019 and TriviaQA (TQA) Joshi et al. 2017 datasets. KPR achieves a substantial 12.6% gain in top 20 accuracy on EQ over the model without KPR extensions and consistently improves performance on the other two datasets. We further build KPR on top of the off-the-shelf bge-base retriever, achieving state-of-the-art results on EQ and TQA compared to other retrievers of similar size. 2 Method KPR is built upon DPR Karpukhin et al. 2020 , a widely used BERT-based dense retriever. We add an attention layer on top of DPR to incorporate entity knowledge from a knowledge base (KB) (see Figure 1 ). We use Wikipedia as the target KB. DPR. DPR encodes a query or passage into a D -dimensional embedding, obtained from the output embedding of BERT’s [CLS] token. Relevance between a query and a passage is scored as the dot product of their embeddings. The model is fine-tuned on datasets containing queries paired with positive and negative passages, using a cross-entropy loss with the dot product scores as logits. See Appendix A for further details. Input entity representation. KPR uses the following two D -dimensional input embeddings: • Entity embedding , assigned to each entity in the KB, represents the entity itself. • Entity position embedding , assigned to each position in the input tokens, encodes positional information. KPR takes the input tokens and a set of entities E=e_{1},\ldots,e_{N} detected from the text using an entity linker. The input representation of each entity is computed by summing its entity embedding and entity position embedding based on its position in the input token sequence. If an entity spans multiple tokens, its position embedding is computed as the average of the embeddings of the corresponding positions Yamada et al. 2020b . To prevent the entity sequence from being empty, a D -dimensional no-op embedding is appended to the sequence of input entity representations. Context-entity attention layer. KPR adopts a single-head key-query-value attention mechanism Vaswani et al. 2017 . The query matrix \mathbf{Q} is computed based on the output embedding of BERT’s [CLS] token (denoted by \mathbf{H}_{\text{{[CLS]}}}\in\mathbb{R}^{1\times D} ). The key and value matrices, \mathbf{K} and \mathbf{V} , are computed based on the input entity representations denoted by \mathbf{U}\in\mathbb{R}^{N+1\times D} : \mathbf{Q}=\mathbf{H}_{\text{{[CLS]}}}\mathbf{X}_{q},\,\,\,\mathbf{K}=\mathbf{U}\mathbf{X}_{k},\,\,\,\mathbf{V}=\mathbf{U}\mathbf{X}_{v}, where \mathbf{X}_{q}\in\mathbb{R}^{D\times D} , \mathbf{X}_{k}\in\mathbb{R}^{D\times D} , and \mathbf{X}_{v}\in\mathbb{R}^{D\times D} are weight matrices. We aim for the attention mechanism to attend to useful entities based on the context by using entity embeddings as the key and the embedding of the [CLS] token, which encodes the context of the input text, as the query. KPR computes the output embedding \mathbf{Z}\in\mathbb{R}^{1\times D} as: \displaystyle\mathbf{Y} \displaystyle=\activation\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{D}}\right)\mathbf{V}, \displaystyle\mathbf{Z} \displaystyle=\layernorm\left(\dropout\left(\mathbf{Y}\right)+\mathbf{H}_{\text{{[CLS]}}}\right), where \activation(\cdot) , \layernorm(\cdot) , and \dropout(\cdot) denote the activation, layer normalization, and dropout functions, respectively. We use the sigmoid function with a length-based bias, \sigmoid(x-\log N+1) Ramapuram et al. 2025 , as the activation. Entity embedder. We compute entity embeddings based on BERT inference over Wikipedia passages that refer to the corresponding entity Ye et al. 2022 . Specifically, for each passage containing an anchor link to the entity, we replace the entity name with BERT’s [MASK] token, extract the output embedding of the [MASK] token, and average the resulting embeddings across passages. The final embeddings are then normalized to match the average norm of BERT’s input token embeddings. Entity linker. We use dictionary-based string matching to detect entities mentioned in the text. The dictionary comprises anchor names and their possible referent entities, obtained from Wikipedia hyperlinks. For ambiguous names (e.g., “Apple,” which can refer to Apple Inc. and Apple (food) ), we do not disambiguate to a single entity, but instead input all possible candidates to maintain recall. Training. The parameters in our attention layer, entity position embeddings, and no-op embedding are initialized randomly. Entity embeddings are kept frozen during training, enabling them to be added or updated dynamically after training. The dropout probability in the attention layer is set to match that of the other layers. We follow DPR Karpukhin et al. 2020 for the remaining training settings including the loss function. Note that entities can be dynamically updated in KPR without retraining, since KPR relies on entity embeddings computed via BERT inference, which are kept frozen during training, and on a dictionary-based entity linker, whose entries can be modified. Computational overhead. KPR adds entity embeddings to BERT, requiring additional storage. However, since the embeddings are used only as input features, they can be implemented as a sparse lookup table and stored not only in GPU memory but also in CPU memory or even on disk, with minimal impact on speed Yu et al. 2025 . Furthermore, KPR introduces only

Read the complete text