Doc-to-LoRA: Learning to Instantly Internalize Contexts

Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange · 2026

Read Doc-to-LoRA: Learning to Instantly Internalize Contexts by Rujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-15902.txt.

Description

Charakorn and colleagues built Doc-to-LoRA, or D2L, a light hypernetwork that learns to distil context in a single forward pass. Long inputs drive in-context learning and document work, but quadratic attention makes inference slow and memory hungry. Distilling each prompt on its own costs too much time and training.

Given a new prompt, D2L makes a LoRA adapter for a target language model. Later queries then run without re-reading the source text, cutting latency and KV-cache memory. On a needle-in-a-haystack task, D2L stores the needle in adapters and nears perfect zero-shot accuracy at lengths over four times the target model native window. On real QA sets with tight compute, it beats plain context distillation while cutting peak memory and update time.

Catalog record

CREATORRujikorn Charakorn, Edoardo Cetin, Shinnosuke Uesaka, Robert T. Lange
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-15902

Browse the full library · About the Tower of Babel