Improving Sparse Memory Finetuning
Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta · 2026
Read Improving Sparse Memory Finetuning by Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05248.txt.
Description
Goyal and three colleagues present a line to refit models with memory modules, giving learning on hardware. Standard methods like full tuning face fatal loss, as changing dense codes causes clash across tasks. Sparse Memory Tuning pins updates to a subset of knobs in open memory layers. They extend with slot choice based on Kullback-Leibler gap, picking odd-token updates against a background spread. Refit models acquire new knowledge with loss of held-out skills, proving sparse updates in practice.
Catalog record
| CREATOR | Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 17802 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.05248