Improving Sparse Memory Finetuning

Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta · 2026

Read Improving Sparse Memory Finetuning by Satyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05248.txt.

Description

Goyal and three colleagues present a line to refit models with memory modules, giving learning on hardware. Standard methods like full tuning face fatal loss, as changing dense codes causes clash across tasks. Sparse Memory Tuning pins updates to a subset of knobs in open memory layers. They extend with slot choice based on Kullback-Leibler gap, picking odd-token updates against a background spread. Refit models acquire new knowledge with loss of held-out skills, proving sparse updates in practice.

Catalog record

CREATORSatyam Goyal, Anirudh Kanchi, Garv Shah, Prakhar Gupta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17802
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05248

Browse the full library · About the Tower of Babel