Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping

Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li · 2026

Read Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mapping by Kaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04308.txt.

Description

Kaustubh Pethkar and three co-authors propose a way to add new knowledge to a language model without retraining its weights, which can erase old knowledge and cannot easily be undone. They ask whether large weight updates are really needed for a small amount of new material.

They model text generation as a Markov process over tokens, where the model's memory is a transition table. Adding knowledge then means extending that table while keeping old transitions intact, which protects what was learned before. They prove a bound on how many samples are needed to learn each new token's behaviour through a dictionary mapping.

Catalog record

CREATORKaustubh Pethkar, Ziyang Xiong, Zuofeng Shang, Yingcong Li
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE80 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04308

Browse the full library · About the Tower of Babel