Doc-to-LoRA: Learning to Instantly Internalize Contexts
Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange · 2026
Read Doc-to-LoRA: Learning to Instantly Internalize Contexts by Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15902.txt.
Description
Charakorn and colleagues built Doc-to-LoRA, or D2L, a light hypernetwork that learns to distil context in a single forward pass. Long inputs drive in-context learning and document work, but quadratic attention makes inference slow and memory hungry. Distilling each prompt on its own costs too much time and training.
Given a new prompt, D2L makes a LoRA adapter for a target language model. Later queries then run without re-reading the source text, cutting latency and KV-cache memory. On a needle-in-a-haystack task, D2L stores the needle in adapters and nears perfect zero-shot accuracy at lengths over four times the target model native window. On real QA sets with tight compute, it beats plain context distillation while cutting peak memory and update time.
Catalog record
| CREATOR | Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-15902