How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment

Rui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu · 2026

Read How to Compress KV Cache in RL Post-Training? Shadow Mask Distillation for Memory-Efficient Alignment by Rui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06850.txt.

Description

Rui Zhu and five co-authors address a memory wall that hits reinforcement learning post-training of language models on long-context reasoning tasks. Online training must first generate exploratory rollouts, and the key-value cache for long contexts consumes enormous memory. Compressing that cache during rollouts helps, but it creates a mismatch: the sampler acts on a sparse context while the learner updates on the full one, and tiny errors amplify badly.

Standard fixes like importance reweighting suffer high variance and poor sample efficiency under that amplified bias. The paper proposes shadow mask distillation as its answer for memory-efficient alignment, targeting the sampler-learner gap directly rather than correcting it statistically after the fact.

Catalog record

CREATORRui Zhu, Weiheng Bai, Qiushi Wu, Yang Ren, Haixu Tang, Yuchu Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06850

Browse the full library · About the Tower of Babel