SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention
Bohan Yu, Wei Huang, Kang Liu · 2025
Read SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention by Bohan Yu, Wei Huang, Kang Liu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06446.txt.
Introduction
Bohan Yu, Wei Huang, and Kang Liu plug live knowledge bases straight into a language model's memory. Retrieval pipelines usually fetch facts with an outside search tool before the model reads them, a chain that can break or lag. SR-KI instead encodes a knowledge base as key-value pairs and loads them into the model's own cache.
Training runs in two stages: first find the model layer best suited to retrieval, then supervise attention at that layer toward the right entries. Retrieval then happens inside the model during a single pass, and updates stay dynamic. The system held forty thousand entries in a seven-billion-parameter model on one graphics card with strong recall.
Catalog record
| CREATOR | Bohan Yu, Wei Huang, Kang Liu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06446
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention Bohan Yu, Wei Huang, Kang Liu Abstract This paper proposes SR-KI, a novel approach for integrating real-time and large-scale structured knowledge bases (KBs) into large language models (LLMs). SR-KI begins by encoding KBs into key-value pairs using a pretrained encoder, and injects them into LLMs’ KV cache. Building on this representation, we employ a two-stage training paradigm: first locating a dedicated retrieval layer within the LLM, and then applying an attention-based loss at this layer to explicitly supervise attention toward relevant KB entries. Unlike traditional retrieval-augmented generation methods that rely heavily on the performance of external retrievers and multi-stage pipelines, SR-KI supports end-to-end inference by performing retrieval entirely within the model’s latent space. This design enables efficient compression of injected knowledge and facilitates dynamic knowledge updates. Comprehensive experiments demonstrate that SR-KI enables the integration of up to 40K KBs into a 7B LLM on a single A100 40GB GPU, and achieves strong retrieval performance, maintaining over 98% Recall@10 on the best-performing task and exceeding 88% on average across all tasks. Task performance on question answering and KB ID generation also demonstrates that SR-KI maintains strong performance while achieving up to 99.75% compression of the injected KBs. Our code will be available at SR-KI . 1 Introduction Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding, analyzing, and generating texts by leveraging their extensive knowledge and powerful reasoning abilities ( 29 ; 21 ; 39 ) . However, in scenarios where users require external knowledge that is not present in or diverges from the information stored in the model’s parameters, efficient and real-time knowledge injection becomes essential. A straightforward approach is to fine-tune the model parameters ( 33 ; 7 ; 3 ) . However, this strategy risks catastrophic forgetting and overfitting by disrupting the original parameter distribution, potentially degrading the model’s pre-existing knowledge. Moreover, full fine-tuning is resource-intensive and inflexible for frequent knowledge updates. Although parameter-efficient tuning methods ( 15 ; 10 ) improve efficiency, they still lack support for continual knowledge updates. Retrieval-Augmented Generation ( 14 ) (RAG) has emerged as a popular alternative, enabling LLMs to access external knowledge by incorporating retrieved content directly into the input prompt. However, RAG follows a pipeline architecture that relies heavily on the performance of external retrievers and is constrained by the limited context window of LLMs ( 37 ; 40 ) . To address the limitations of retrieval-based methods, recent long-context LLMs ( 21 ; 8 ) extend the context window, enabling direct reasoning over the entire input. However, this comes at the cost of significant computational and memory overhead ( 4 ) , limiting their scalability. KBLaM ( 32 ) offers an alternative by injecting external knowledge into the KV cache ( 22 ) through projection adapters, allowing the model to attend to key-value representations rather than store specific facts. Nevertheless, as the scale of injected knowledge increases, KBLaM fails to focus on the most relevant information, resulting in severe performance degradation. Moreover, all of these approaches remain challenging to attribute the model’s output to the injected knowledge, raising concerns about controllability and interpretability ( 17 ; 12 ; 1 ) . In this paper, we propose SR-KI ( S calable and R eal-Time K nowledge I ntegration into LLMs via Supervised Attention), a novel method for injecting external knowledge into LLMs dynamically via a supervised attention mechanism. Similar to KBLaM, SR-KI transforms unstructured factual knowledge into structured knowledge bases (KBs) of ( subject, relation, object ) triples and injects them into the latent space of LLMs. Specifically, SR-KI converts each knowledge triple into a key-value pair, using the subject and relation as the key and the object as the value. The key and value are independently embedded into a vector pair using a pretrained sentence encoder, and then projected through learnable single-layer adapters to match the embedding dimension of the LLM’s KV cache, enabling their injection into the attention mechanism as external key-value pairs. This design allows the model to learn generalizable key-value mappings rather than memorize specific facts, while keeping attention computation linearly scalable with the number of triples. Prior studies have discovered specific layers in LLMs that are particularly sensitive to knowledge injection ( 17 ; 31 ) . In our study, we further find that this attributes to the model’s architecture rather than task-specific factors. Inspired by this observation, a two-stage training paradigm is adopted. The first stage involves identifying the retrieval layer—a critical point where knowledge injection has the greatest influence. In the second stage, an attention-based loss is introduced at this layer to enable multi-objective optimization, explicitly guiding the model to focus on the most relevant KB entries. By directly supervising the attention behavior at this layer, our method equips it with the ability to precisely attend to pertinent knowledge even under large-scale injection. This mechanism allows SR-KI to effectively prune irrelevant knowledge to reduce inference latency and memory usage, while simultaneously enabling the reuse of critical KBs in subsequent layers to promote more efficient and coherent knowledge utilization. All these processes are performed in an end-to-end manner, with knowledge integration and response generation jointly executed in a single forward pass without relying on external modules or multi-stage pipelines. Beyond efficient knowledge access, SR-KI further enables knowledge reference and traceability by guiding the model to generate both factual content and its corresponding source. This design meets the growing demand for transparent and verifiable outputs in knowledge-intensive tasks ( 12 ) . To support this, the Reference ID KB is introduced, where each knowledge triple is assigned a randomly generated uppercase ID. These reference triples are encoded and injected into the LLM following the same key-value format as factual knowledge, allowing the model to jointly predict the knowledge and its associated ID, thereby achieving factual grounding and source attribution. SR-KI supports real-time and large-scale knowledge injection by aggressively compressing 40K KBs on a single A100 40G GPU down to the top-100 during inference, achieving up to 99.75% compression while maintaining strong reasoning and retrieval performance. Even as the KB size scales to 40K, SR-KI sustains over 95% Recall@100 and 88% Recall@10, demonstrating robust retrieval capabilities. On question answering tasks, it achieves consistently high performance under large-scale settings, with a performance margin of up to 70 points compared to baseline levels observed in same settings. In conclusion, our key contributions are as follows: • We propose SR-KI, a two-stage training framework that employs supervised attention for efficient and dynamic knowledge injection into LLMs. By leveraging learnable KV projection adapters and attention-guided supervision, SR-KI enables scalable, precise, and minimally invasive integration of structured external knowledge. • SR-KI incorporates a dedicated retrieval layer that enables accurate and efficient selection from large-scale KBs, achieving up to 99.75% compression while preserving strong task performance and retrieval effectiveness. • SR-KI jointly generates the knowledge and its source KB ID, enabling transparent and verifiable outputs. 2 Related Works Retrieval-Augmented Generation (RAG) Retrieval-Augmented Generation ( 14 ) enables LLMs to access external knowledge by retrieving relevant passages and appending them to the input context. While effective, RAG is limited by the context window size and the quadratic attention cost of transformers