Doc-to-LoRA: Learning to Instantly Internalize Contexts Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange Abstract Long input sequences are central to in-context learning, document understanding, and multi-step reasoning of Large Language Models (LLMs). However, the quadratic attention cost of Transformers makes inference memory-intensive and slow. While context distillation ( † † affiliationnotice: *S. Uesaka contributed while he was an intern at Sakana AI. ) can transfer information into model parameters, per-prompt distillation is impractical due to training costs and latency. To address these limitations, we propose Doc-to-LoRA ( ), a lightweight hypernetwork that meta-learns to perform approximate within a single forward pass. Given an unseen prompt, generates a LoRA adapter for a target LLM, enabling subsequent queries to be answered without re-consuming the original context, reducing latency and KV-cache memory consumption during inference of the target LLM. On a long-context needle-in-a-haystack task, successfully learns to map contexts into adapters that store the needle information, achieving near-perfect zero-shot accuracy at sequence lengths exceeding the target LLM’s native context window by more than 4\times . On real-world QA datasets with limited compute, outperforms standard while significantly reducing peak memory consumption and update latency. We envision that can facilitate rapid adaptation of LLMs, opening up the possibility of frequent knowledge updates and personalized chat behavior. Checkout our project at github.com/SakanaAI/doc-to-lora . 1 Introduction LLMs are commonly adapted to documents, tasks, and user preferences by placing relevant information in the context window, also known as prompting or in-context learning ( Brown et al. 2020 , ICL, ) . While effective and convenient, ICL is transient and memory-intensive; long prompts increase latency and memory via quadratic attention and KV-cache growth. Moreover, generation quality typically degrades under longer context lengths ( Liu et al. 2024 ; Li et al. 2025 ; Hong et al. 2025 ) . A standard remedy is to compress information from context into model parameters via supervised finetuning ( Zhang et al. 2023 ; Pareja et al. 2025 , SFT, ) . However, SFT requires collecting a task-specific dataset that represents the desired behavior and risks overfitting when data is scarce. Furthermore, repeated expensive SFT processes are required if the information, e.g., user preference, changes over time. These constraints limit the possibility of scalable model customization by practitioners. Context distillation ( ) offers a compelling alternative “citepaskell2021contextdistillalignment,choi2023fixed,padmanabhan2023propagating,snell2023contextdistill,bhargava2024promptbaking,caccia2025knowledge_modules,eyuboglu2025cartridges,kujanpaa2025knowledgeinjection,qi2025incontextedit,shin2025generativepromptinternalization. At its core, trains an LLM—without access to relevant information—to imitate its own outputs when the information is provided in context. Once the information is internalized 1 1 1 We define internalization concretely in Section 2 . Informally, internalization is a process that transforms information into a model’s parameters such that the model can later access it. , subsequent inference calls are significantly faster as the model does not need the information to be in its context window. Still, its training is slow and memory-intensive, making it impractical for interactive or on-device applications. Furthermore, when the information sources are constantly changing, repeated slow processes must take place. Figure 1 : An overview of training (left) and downstream performance (right). learns to efficiently internalize information, outperforming traditional CD while significantly reducing latency and memory consumption across question-answering benchmarks under limited query budgets. In this work, we aim to combine the convenience of ICL with the effective internalization of . We propose to meta-learn the process into a hypernetwork ( Ha et al. 2016 ) . In other words, we meta-train the hypernetwork to represent a mapping from a given context to its corresponding weight updates produced by . Conceptually, after reading a context once, a lightweight hypernetwork generates a context-specific LoRA adapter ( Hu et al. 2022 ) . With the adapter, the target LLM can respond to subsequent queries without the context, reducing latency and KV-cache size. Once trained, the hypernetwork can be reused for any new context. A single, inexpensive forward pass executes the learned process, enabling low-latency internalization. We call this method Doc-to-LoRA ( ). To enable to internalize contexts longer than those seen during training, we base our hypernetwork on the Perceiver architecture ( Jaegle et al. 2021 ) , allowing it to map variable-length inputs to a fixed-shape LoRA adapter. Furthermore, utilizes a chunking mechanism that allows it to produce higher-rank LoRA adapters for longer contexts without changing the output shape of the hypernetwork. This mechanism is crucial for processing documents longer than the context window of the target LLM. As a result, on a synthetic Needle-in-a-Haystack (NIAH) task, achieves near-perfect zero-shot accuracy on contexts beyond 32 K tokens, despite being trained only on sequences of up to 256 tokens, indicating that is capable of generalizing beyond the lengths of its training data and that the chunking mechanism is effective at processing long sequences. Empirically, on real-world QA tasks, learns an effective mapping between contexts and their corresponding LoRA adapters, outperforming under low compute budgets while significantly reducing internalization latency and compute cost. On long-document QA tasks, generalizes in a zero-shot manner to documents exceeding the training length, without being explicitly trained to internalize such long inputs. A conceptual overview of is shown in Section 1 . We summarize our contributions as follows: • We introduce a meta-training objective that distills the process into a hypernetwork. This approach amortizes the overhead associated with traditional by training the hypernetwork to emulate the entire process in a single forward pass ( Section 3.1 ). • We introduce a well-designed architecture that makes the hypernetwork robust to varying input lengths and allows it to output higher-rank LoRAs by chunking long inputs ( Section 3.2 ). Consequently, can internalize needle information at near-perfect zero-shot accuracy up to 4\times the maximum length of the base LLM on an NIAH task ( Section 4 ). • We empirically validate and show that, under limited model update budgets, it outperforms with improved internalization efficiency, significantly reducing update latency and memory usage. Furthermore, generalizes zero-shot to long-document QA tasks with even greater computational efficiency compared to the gains observed on shorter contexts ( Section 5.1 ). Remarkably, zero-shot transfers visual information effectively from a visual-language model (VLM) to a text-based LLM, allowing the target model to classify images purely through internalized information ( Section 5.2 ). Finally, analyses of extreme generalization, performance gains, ablations, and failure modes of are included in Section 6 and App. C . 2 Preliminaries Context Distillation ( ) is a self-distillation method that internalizes behaviors induced by an in-context prompt, c , into a model’s parameters. Unlike traditional knowledge distillation ( Buciluǎ et al. 2006 ; Hinton et al. 2015 ) , uses the same LLM for both the “teacher” and the “student.” The teacher has access to the prompt c , while the student does not. Concretely, given a context and query pair (c,x) , we sample a response y from the teacher, parameterized by \theta : y\sim p_{\theta}(\cdot\mid x,c) . Then, y is used as the target for training the student with only access to the query x . Mathematically, the training objective can be written as \displaystyle\min_{\theta_{c}}\;\;\Big[\mathrm{KL}\big(p_{\theta}(y\mid x,c)\;\|\;p_{\theta_{c}}(y\mid x)\big)\Big], (1) where \mathrm{KL}(\cdot\|\cdot) denotes the Kullback–Leibler divergence between two probability distributions and the context-specific parameters \theta_{c} are initialized from the original parameters \theta . Given that modern LLMs are effective at following instructions provided in context, is a practical way to internalize new knowledge or behaviors without expensive data gathering steps needed for SFT. Defining Internalization. Optimizing Equation 1 risks overfitting since it only considers a single (c,x,y) triplet. We refer to it as query-dependent distillation as the learning signal comes solely from a single query x . A more robust alternative is query-independent distillation, which utilizes multiple queries, X=\{x_{i}\}_{i=1}^{n} , and responses, Y=\{y_{i}\sim p_{\theta}(\cdot|x_{i},c)\}_{i=1}^{n} . Collectively, X and Y make a small dataset \mathcal{D}_{c}=\{(x_{i},y_{i})\}_{i=1}^{n} specifically for internalizing context c with various queries and self-generated responses. Typically, X is generated by prompting an LLM to generate relevant questions for the context c ( Caccia et al. 2025 ; Kujanpää et al. 2025 ; Eyuboglu et al. 2025 ) . Thus, the query-independent distillation objective becomes \displaystyle\min_{\theta_{c}}\;\;\mathbb{E}_{(x,y)\sim\mathcal{D}_{c}}\Big[\mathrm{KL}\big(p_{\theta}(y\mid x,c)\;\|\;p_{\theta_{c}}(y\mid x)\big)\Big] (2) We define the optimization in Equation 2 as the internalization process of a context c into model parameters \theta . Successful internalization means the model can access information from c through the internalized parameters \theta_{c} , behaving as if c were provided in context. Context distillation has strong implications for real-world applications as it enables the creation of many specialized models without requiring explicit data collection. For instance, persistent instructions like safety and alignment prompts are prime candidates for internalization. These prompts are often tailored for different use cases and must remain in the context window throughout deployment. 3 Meta-Learning Context Distillation 3.1 Learning to Internalize Context with Hypernetworks In this work, we focus on meta-learning query-independent , which allows the internalized parameters \theta_{c} to be used with unseen downstream queries. Thus, the meta-training phase trains a hypernetwork to map a context c to a set of adapter parameters that modify a frozen base model \theta , yielding a context-internalized model \theta_{c}=\theta+\Delta W_{c} , where \Delta W_{c}=H_{\phi}(c) and H_{\phi} represents a mapping from a raw string to LoRA parameters, parameterized by \phi ( Section 3.2 ). The overall meta-training process is similar to the vanilla (Eq. 2 ) with one key distinction: Optimizing a single H_{\phi} that generalizes across many contexts (tasks), rather than optimizing separate \Delta W per context. The hypernetwork must be exposed to a vast number of contexts to robustly represent the process. Consequently, the meta-training dataset \mathcal{D} must include diverse contexts, queries, and responses: \mathcal{D}=\{c_{i},\mathcal{D}_{c_{i}}\}_{i=1}^{n} . Mathematically, we optimize H_{\phi} to minimize the divergence between the context-conditioned teacher and the context-internalized student: \displaystyle\min_{\phi}\;\;\mathbb{E}_{(c,\mathcal{D}_{c})\sim\mathcal{D}}\;\mathbb{E}_{(x,y)\sim\mathcal{D}_{c}}J(x,y,c), (3) \displaystyle J(x,y,c)=\mathrm{KL}\big(p_{\theta}(y\mid x,c)\;\|\;p_{\theta+H_{\phi}(c)}(y\mid x)\big) (4) After training on a large corpus \mathcal{D} , the trained hypernetwork should serve as a generic mapping between context information c and corresponding internalized parameters \theta_{c} , generated by a single hypernetwork forward pass. That is, a trained mapping H amortizes both the query generation process and the backpropagation needed by traditional . The data generation pipeline is described in App. B.1 . 3.2 Architecture Next, we describe how maps a context string into LoRA matrices (Fig. 1 shows the overview of this process). A context c is fed through the frozen target LLM to obtain per-layer token activations Z\in\mathbb{R}^{L\times N\times D} , where L is the number of Transformer layers (including the embedding layer), N is the number of context tokens, and D is the hidden size. We denote the activations at depth l by Z_{l}\in\mathbb{R}^{N\times D} , with Z_{0} being the token embeddings. For each transformer layer l\in\{1,\dots,L\} , a shared hypernetwork h_{\phi} consumes Z_{l-1} and outputs low-rank LoRA parameters ( Hu et al. 2022 ) : h_{\phi}(Z_{l-1})=\Delta W_{l}=B_{l}A_{l} . Specifically, each target weight W_{l}\in\mathbb{R}^{d^{\text{out}}_{l}\times d^{\text{in}}_{l}} is adapted as \displaystyle W^{\prime}_{l}\;=\;W_{l}\;+\;\alpha_{l}\,B_{l}A_{l};\;A_{l}\in\mathbb{R}^{r\times d^{\text{in}}_{l}},\;B_{l}\in\mathbb{R}^{d^{\text{out}}_{l}\times r}, (5) where r is the rank and \alpha_{l} is a learnable per-layer scalar. Perceiver-Based Hypernetwork: We structure the hypernetwork h_{\phi} as a Perceiver-style cross-attention module ( Jaegle et al. 2021 ) that maps variable-length inputs ( Z_{l-1} ) to a fixed number of latent queries (the LoRA rank). We briefly describe the simplest configuration of the Perceiver with a single cross-attention block here. 2 2 2 The cross-attention operator can be repeated many times and interleaved with latent self-attention blocks. Specifically, for each layer l , we use r learnable, input-independent latent queries Q_{m}\in\mathbb{R}^{r\times d_{q}} . Cross-attending Q_{m} to Z_{l-1} yields r latent vectors: \displaystyle U_{l}\;=\;\mathrm{XAttn}\big(Q_{m},\,K(Z_{l-1}),\,V(Z_{l-1})\big)\in\mathbb{R}^{r\times d_{u}}, (6) where K(\cdot) and V(\cdot) are linear projections. Two per-layer output heads then produce the LoRA matrices by mapping each latent to a row of A_{l} and a column of B_{l} . Overall, the mapping from a context string to LoRA matrices, H_{\phi} , given context c and its token activations Z , can be written as \displaystyle H_{\phi}(c)\;=\; \displaystyle\Delta W\;=\;\{\Delta W_{l}\}_{l\in\{1,\dots,L\}}, (7) \displaystyle\Delta W_{l}\;=\;B_{l}A_{l}\;=\;h_{\phi}(Z_{l-1}) (8) Long-Context Composition via Chunking: For long contexts, we partition c into K contiguous chunks \{c^{(k)}\}_{k=1}^{K} and process each chunk independently through the hypernetwork, producing per-chunk adapter (A^{(k)}_{l},B^{(k)}_{l}) . We then combine chunks by concatenating along the rank dimension \displaystyle A_{l}\;=\;\begin{bmatrix}A^{(1)}_{l}\\ \vdots\\ A^{(K)}_{l}\end{bmatrix},\qquad B_{l}\;=\;\big[B^{(1)}_{l}\;\cdots\;B^{(K)}_{l}\big], (9) resulting in LoRA matrices with the total rank r\cdot K . This composition allows to integrate information across many chunks without changing the output shape of the hypernetwork. Note that the described architecture is not limited to generating LoRA matrices. In App. D , we empirically show that it can effectively learn to output compressed KV cache ( Li & Liang 2021 , i.e., prefix-tuning, ) . 4 Implanting Synthetic Needle-in-a-Haystack Information In this section, we aim to show that (i) successfully induces knowledge internalization, enabling the base model to recall the implanted information without reading the raw context, (ii) effectively bypasses the inherent context-length limitations of the base language model, and (iii) reduces the computational requirements for inference, especially when the inputs are long. For illustration purposes, we evaluate on a synthetic needle-in-a-haystack (NIAH) information retrieval task. Our needle-in-a-haystack (NIAH) task is primarily based on the RULER benchmark ( Hsieh et al. 2024 ) . Briefly, the NIAH task requires the model to locate a specific piece of information (needle) within a long, distracting document (haystack). The needle is a sentence defining a special 4-digit number, e.g., “ The special magic number is 0042. ” This sentence is randomly inserted into a document filled with distractor text. The goal is to accurately retrieve the number when prompted. We use gemma-2-2b-it ( Team et al. 2024 ) with 8\text{K} context length as the base LLM in all experiments. During ’s meta-training, we use input contexts ranging from 32 to 256 tokens in length. The training inputs are randomly chunked from 1 to 8 chunks with a minimum chunk size of 25 tokens. We use a simplified training setup for this pedagogical experiment (see App. A for more details). During evaluation, the baseline has direct access to both the haystack and the query. For , the base LLM does not have direct access to any part of the original context but is simply given the query prompt: “ What is the special magic number? Reply with only the number. ” To perform well at this task, must learn to map context information into a LoRA adapter that stores the value of the needle. With successful internalization, the adapted base model would be able to give a correct response based purely on the knowledge contained in the LoRA adapter. processes the inputs by segmenting them into equal-sized chunks with 1024 tokens as the maximum chunk size. This chunk size is four times larger than the 256-token maximum sequence length seen during training. (a) (b) Figure 2 : NIAH retrieval performance (top) and additional memory needed for inference (bottom). Learns Effective for NIAH Task and Generalizes Beyond Base Model’s Context Length: The result in “cref fig:niah˙perf˙vs˙ctx˙len˝ (left) confirms the effectiveness of our approach. “ours successfully internalizes the needle information and achieves perfect accuracy similar to the base model with in-context information up to 8K tokens. When the haystack is longer than 8K tokens, the base model’s performance drops sharply due to its limited context length. In contrast, maintains high retrieval accuracy across these longer sequences. Notably, performance remains close to perfect up to 40 chunks (40K tokens), which is quintuple the number of chunks the model has been exposed to during its training phase. Beyond this point, performance begins to degrade gracefully. These results demonstrate that exhibits strong generalization capabilities for both chunk size and the total number of chunks. Reduces Inference Cost Under Long Inputs: The result in “cref fig:niah_perf_vs_ctx_len (right) shows that not only achieves high accuracy but also demonstrates significant efficiency improvements, requiring less memory than the base model, particularly at extended context lengths. The base model uses more than 12 GB of additional memory to generate a response to a 128K-token haystack. In contrast, the model with internalized knowledge consistently uses significantly lower memory ( < 50 MB) regardless of the length of the haystack. This result highlights a potential real-world application, where users first internalize long private documents, avoiding memory-intensive KV cache at inference. 5 Experiments In this section, we move from the synthetic NIAH task to a more realistic setting, where has to learn to approximate generic from a large corpus of English documents. We evaluate the ability of to operate as a context internalizer for question-answering tasks on 6 real-world benchmarks, including short-passage and long-document QA tasks. A key advantage of on these tasks is its ability to provide instant and inexpensive internalization, which allows the base LLM to answer subsequent queries without consuming the context window. Experimental Setup: We use the data generated by the process described in App. B.1 to meta-train . Our Perceiver-based has 8 cross-attention blocks without self-attention layers. It splits the inputs into equal-sized chunks with 8K tokens and outputs a rank-8 LoRA adapter for each context chunk. Each generated adapter is applied to the “down projection” layer of each MLP block of the base model. In total, it has only 309 M trainable parameters. During evaluation, can operate in batched and iterative LoRA generation. In batched mode, it batches the token activations {Z_{l}} across all layers, producing LoRA adapters for all the layers within a single forward pass. Iterative mode, on the other hand, produces the adapter one layer at a time. The two modes allow us to prioritize either speed (batched) or lower memory consumption (iterative). 3 3 3 Both modes are mathematically equivalent. Any task performance differences are caused by different low-level matmul kernels and the non-commutative nature of floating-point operations. We consider the following in-parameter knowledge baselines—where the model has to answer input queries based on its internal knowledge: (i) (oracle) serves as the empirical upper-bound of in-parameter knowledge methods as it optimizes the internalized weights directly on the target query (Eq. “ref eq:cd-oracle-obj), (ii) trains the internalized weights using generated queries from “texttt gemma-3-12b-it (Eq. 2 ), (iii) T2L ( Charakorn et al. 2025 ) is a hypernetwork-based method that directly maps the context to LoRA, similar to . However, it has been trained on SFT datasets with next-token prediction loss on ground-truth tokens, and (iv) Base model (without context) serves as the lower bound, as it does not have access to any context information. Both baselines internalize the context into a rank-8 LoRA adapter applied at the “down projection” layer. We use the publicly available checkpoint for T2L, which has been trained to output rank-8 adapters for the K and V projection layers. To better contextualize the reported performance, we also include in-context knowledge baselines—where the model can simply retrieve the answer directly from the provided context: (i) Base model with context serves as the upper-bound as it can directly access the answer in context, and (ii) LLMLingua-2 ( Pan et al. 2024 ) , a prompt compression method. Results of other base models ( Mistral-7B-Instruct-v0.2 and Qwen3-4B-Instruct-2507 ) are in App. E . 5.1 Question-Answering from Implanted Knowledge The following experiments test ’s ability to accurately store and retrieve information effectively from generated LoRA adapters. We report the test performance on unseen instances and use word-level ROUGE-L F1 scores as the main task performance metric. The reported performance is relative to the base model with direct access to the contexts. We provide its absolute performance in Table 11 . 5.1.1 Efficient and Effective Internalization on Reading Comprehension Tasks Figure 3 : QA performance on SQuAD compared to the used context length ratio (left), update latency (middle), and additional memory needed for model updates (right). LLMLingua-2 compresses the input with [ 10\%,20\%,40\%,60\%,80\%,90\% ] compression rates from right to left (gray dots). We assess performance on three standard reading comprehension benchmarks: SQuAD ( Rajpurkar et al. 2016 , span extraction, ) , DROP ( Dua et al. 2019 , discrete reasoning over passages, ) , and ROPES ( Lin et al. 2019 , reasoning with background knowledge, ) . As shown in Figure 3 (left), outperforms all the in-parameter baselines across all three benchmarks (see more results in App. C ), achieving 82.5\% relative performance compared to the ICL upper bound on the SQuAD benchmark. Compared to LLMLingua-2, performs roughly similarly to compressing the context down to 40\% of its original length, but it does so while removing the context entirely. The main benefit of using is its update efficiency. not only outperforms the baseline given limited compute and time constraints, it significantly reduces update latency and memory ( Figure 3 , middle and right). Specifically, instantly internalizes context information within less than a second using either batched or iterative mode. (oracle) uses around 40 seconds to internalize information due to repeated stochastic gradient descent updates. Vanilla requires even more time to internalize because of the additional query generation process, totaling more than 100 seconds. While T2L can instantly update the model, it cannot effectively internalize knowledge due to its highly focused training data. requires low additional memory during the internalization process, similar to that of (oracle). Both use less than 2 GB of VRAM during the update process across the three benchmarks. In contrast, with generated queries uses more than 40 GB due to backpropagating on many queries at once. This result highlights the main practical benefit of : providing effective instant internalization with low memory requirement. Due to the space limit, we provide results on DROP and ROPES benchmarks in App. C . The overall trends and observations are similar to the results on the SQuAD dataset shown in Figure 3 . In the next experiment, the memory efficiency of is much more impactful when considering longer context tasks. 5.1.2 Instant Zero-Shot Internalization of Long-Context Information Table 1 : Performance, update memory, and latency of in-parameter knowledge methods on 2WikiMultihopQA benchmark. Method Normalized Performance ( \uparrow ) Additional Update Memory (GB, \downarrow ) Mean Update Latency (s, \downarrow ) (oracle query) 0.901 7.820 40.171\pm 0.351 (batched) \mathbf{0.857} 11.522 \mathbf{0.209}\pm 0.123 (iterative) 0.844 \mathbf{3.791} 0.551\pm 0.101 (25 generated queries) {0.745} {59.925} 465.454\pm 67.868 (5 generated queries) 0.704 79.371 72.537\pm 7.821 Figure 4 : Long document QA performance. LLMLingua-2 compresses the input with [ 20\%,40\%,60\%,80\%,90\% ] compression rates from right to left (gray dots). We extend our evaluation to long-context scenarios, which pose a significant challenge for standard due to memory and computational constraints of distilling long context, using three document-grounded QA benchmarks from LongBench dataset ( Bai et al. 2023 ) , 2WikiMultihopQA ( Ho et al. 2020 ) , MultiFieldQA , and QASPER ( Dasigi et al. 2021 ) . The length of test samples can go up to 32 K tokens. We note that has never seen such long sequences during training. Specifically, the longest training sample is 2{,}344 token long (see Appendix B ). In this experiment, uses generated queries from truncated documents since the base model performs worse without truncation. The results in Section 5.1.2 and Figure 4 show that can effectively internalize long-context documents without being explicitly trained to do so. Like the previous experiment, outperforms with generated queries and almost reaches the upper-bound performance of the oracle on 2WikiMultihopQA. Even with 5 queries, uses up to 79 GB of VRAM to internalize the documents. Furthermore, the oracle requires more than 7 GB of VRAM during the update. In contrast, with iterative LoRA generation uses 2 x less memory compared to the oracle’s update while maintaining sub-second internalization. Results on other benchmarks are included in App. C . Once again, this result underscores the practical benefits of for real-time and on-device applications. In this experiment, we also investigate the additional memory used during response generation of the base LLM since it requires a non-negligible amount of VRAM to read long-context information. Figure 4 shows that answers more accurately than the baseline while significantly reduces memory used during response generation. Specifically, the ICL baseline requires around 1 GB of VRAM while all in-parameter knowledge methods require less than 100 MB. Interestingly, on 2WikiMultihopQA and MultiFieldQA datasets, we observe that after internalizing a long context via , the performance of the LLM slightly improves when reading the context directly again (indicated by Ours + truncated context in Figure 4 ). Also, when used with LLMLingua-2 with specific compression rates, the performance improves over feeding the base model with uncompressed truncated inputs. We hypothesize that this might be because of the lost-in-the-middle ( Liu et al. 2024 ) or attention noise ( Ye et al. 2025 ) phenomena. Intuitively, text-based compression reduces unnecessary words from the context, and, therefore, can reduce attention noise and improve performance. While produces similar improvement, we believe that the improvement comes from different mechanisms. When the model faces strong attention noise or the answers are truncated, the LLM then falls back on using its internal knowledge ( Tao et al. 2024 ) , which has been updated by . We do not observe this effect in any of the benchmarks used in the previous experiment, presumably because attention noise does not occur under shorter context lengths. 5.2 Zero-Shot Internalization of Visual Information Table 2 : Using a VLM as the context encoder. SQuAD DROP ROPES Imagenette (LLM \Rightarrow LLM) 0.814 0.655 0.906 N/A (VLM \Rightarrow LLM) 0.705 0.568 0.772 75.03% Since is based on the Perceiver architecture that can map arbitrary-length inputs to fixed-size outputs, we are not restricted to using the target LLM as the context encoder. In this experiment, we explore the feasibility of using the proposed architecture for bridging between a VLM ( gemma-3-4b-it ) and a pure-text model ( gemma-2-2b-it ). We want to see whether , without seeing any images during training , can internalize visual information from the VLM zero-shot. Since the number of layers of the VLM and the LLM are not the same, maps the activations from the first 26 layers of the VLM to the corresponding layers of the target LLM. We keep the rest of the training setup identical to the main experiment. Only the language modeling part of the VLM is used during training. To test whether the target model can classify images based solely on the internalized information, we use the Imagenette dataset ( Howard 2019 ) , a 10-class subset of ImageNet ( Russakovsky et al. 2015 ) . The target model is prompted with the following text: “ What is in this image? Choose exactly one of the following classes: tench, English springer, cassette player, chainsaw, church, French horn, garbage truck, gas pump, golf ball, parachute. Respond with only the correct class without any other text. ” Achieving better-than-random accuracy ( 10\% ) on this task means that can zero-shot transfer visual information to a text-only LLM. The results are shown in Section 5.2 . successfully learns to map the VLM’s activations into LoRA matrices of the target LLM. Although using a VLM as the context encoder negatively impacts text-based QA performance, it enables us to directly communicate visual information extracted by the VLM into the parameters of the target text-only model. Specifically, the target model gets 75.03\% accuracy purely through the internalized information. This is a striking result given that and the target LLM have never seen information from any other modalities except text. 6 Analyses Zero-Shot Query Internalization: To further test the robustness of D2L, we use the existing SQuAD dataset with a slight modification. Instead of internalizing the document, we use D2L to internalize the query. This means that, during evaluation, the target model sees the document directly but not the query. This experiment represents an extreme generalization test for , as it is trained to internalize knowledge, not queries. The results in Section 6 are encouraging and suggest that D2L can function properly under such an extreme generalization test. In this “swapped” configuration, while performs worse, it still achieves moderate performance on the ROUGE-L recall metric (0.587) and outperforms the no-context baseline (0.185). We observe that the query-internalized model occasionally generates correct answers, but its outputs are typically verbose, leading to a sharp decrease in precision. This test suggests that could be used to internalize other kinds of information beyond factual information from documents. Qualitative examples of this behavior are provided in Figure 8 . Table 3 : SQuAD query internalization. Method Recall Precision Base model w/ context 0.886 0.876 0.740 0.720 (swapped) {\pagecolor[HTML]{d3ebd8}0.587} {\pagecolor[HTML]{d3ebd8}0.044} Base model w/o context 0.185 0.205 Table 4 : SQuAD (100 samples) performance with varying queries. Method Normalized Performance Update Latency (s) (oracle) {0.988} {8.763\pm 0.313} 0.866 {\textbf{0.086}\pm 0.061} (100 generated queries) {0.650} {631.101\pm 12.879} (50 generated queries) {0.601} {311.661\pm 7.211} (20 generated queries) {0.506} {129.044\pm 5.813} Emulates over Many Generated Queries: In the main experiments (“cref sec:experiments˝), “ours consistently outperforms vanilla under small query budgets. We hypothesize that performs better because the hypernetwork learns to emulate the effect of distilling over a large number of queries per context. Although each training example provides only 10 generated queries, training across millions of context samples exposes the hypernetwork to a larger and more diverse query distribution. In other words, the training samples collectively regularize to be robust to a wider variety of queries than presented in any single sample. Empirically, increasing the number of generated queries for vanilla improves its performance, but it remains well below that of ( Section 6 ). On a 100-sample subset of SQuAD, its performance rises from 0.506 (20 queries) to 0.650 (100 queries). Crucially, with 100 queries takes more than 10 minutes to internalize each sample . In contrast, achieves 0.866, substantially closer to the oracle CD upper bound, without incurring per-sample query generation or backpropagation. Although increasing the budget to hundreds or thousands of queries would likely boost performance beyond that of , the update latency would also increase substantially. Such a large latency would render the user experience non-responsive, especially in fast-changing environments, e.g., active codebases or agentic tasks. This result highlights the effectiveness of at instantly internalizing knowledge in the sub-second regime. This trend supports the interpretation that the hypernetwork has learned a mapping that approximates the process with many more queries per context. A complementary explanation is that has learned a specialized but biased form of . Specifically, might assume that subsequent queries will always be related to the internalized knowledge ( Section C.3 ). ’s training curves and ablation studies on training data sources and LoRA rank can be found in App. C . 7 Related Work Hypernetworks have been used to adapt LLMs on-the-fly for different use cases ( Ivison & Peters 2022 ; Ivison et al. 2023 ; Phang et al. 2023 ; Lv et al. 2024 ; Charakorn et al. 2025 ) . Notably, MEND ( Li et al. 2024 ) trains a hypernetwork via the objective to compress few-shot examples into prefix tokens ( Li & Liang 2021 ) . Similarly, Gisting ( Mu et al. 2024 ) uses the objective to train “gist” tokens that compress task instructions. Cartridges ( Eyuboglu et al. 2025 ) aims to utilize abundant sleep-time compute budget for based on the prefix-tuning parameterization. In contrast to prior works, aims to capture a generic process that can be applied to arbitrary information presented as context while being time- and memory-efficient. A closely related method, Generative Adapter ( Chen et al. 2025 , GA, ) , optimizes a hypernetwork using the next-token prediction loss on ground-truth tokens. GA first trains its hypernetwork on a pre-training corpus and then finetunes it on curated queries and responses from existing SFT datasets. In contrast, meta-trains the hypernetwork with the objective—using primarily generated queries and self-responses—to output a context-specific LoRA adapter. Empirical results in App. C.2 also suggest that using the objective is crucial for robust generalization of the hypernetwork. We also note that GA achieves a significantly higher ROUGE-L F1 score than the base model (see Appendix E ) because of its training method that directly trains the model to output the ground truth tokens via the SFT loss. However, it achieves much lower ROUGE-L recall, indicating that the improved F1 scores comes solely from the fact that Generative Adapter outputs much shorter responses despite answering with less factual accuracy (represented by the ROUGE-L recall score). Finally, the use of generated queries and self-responses allows straightforward extensions of to domains where finetuning datasets are not available. Due to the space constraint, extended Related Work and Conclusion sections are provided in App. F and G .