Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping
Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li · 2026
Read Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping by Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04308.txt.
Description
Kaustubh Pethkar and three co-authors propose a way to add new knowledge to a language model without retraining its weights, which can erase old knowledge and cannot easily be undone. They ask whether large weight updates are really needed for a small amount of new material.
They model text generation as a Markov process over tokens, where the model's memory is a transition table. Adding knowledge then means extending that table while keeping old transitions intact, which protects what was learned before. They prove a bound on how many samples are needed to learn each new token's behaviour through a dictionary mapping.
Catalog record
| CREATOR | Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04308