How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment
Rui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu · 2026
Read How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment by Rui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06850.txt.
Description
Rui Zhu and five co-authors address a memory wall that hits reinforcement learning post-training of language models on long-context reasoning tasks. Online training must first generate exploratory rollouts, and the key-value cache for long contexts consumes enormous memory. Compressing that cache during rollouts helps, but it creates a mismatch: the sampler acts on a sparse context while the learner updates on the full one, and tiny errors amplify badly.
Standard fixes like importance reweighting suffer high variance and poor sample efficiency under that amplified bias. The paper proposes shadow mask distillation as its answer for memory-efficient alignment, targeting the sampler-learner gap directly rather than correcting it statistically after the fact.
Catalog record
| CREATOR | Rui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06850