EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KL
Lunjun Zhang, Jimmy Ba · 2026
Read EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KL by Lunjun Zhang, Jimmy Ba (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04417.txt.
Description
A moving anchor steadies policy gradient training for language models. Lunjun Zhang and 1 co-author propose EMA Policy Gradient in a 2026 paper. Standard methods anchor updates to a fixed reference policy; theirs uses an exponential moving average of the policy itself, like a target network in value learning.
A top-k KL estimator joins the anchor, interpolating between exact and sampled divergence while staying unbiased in both value and gradient. The authors derive stability conditions for the moving anchor. Combined with GRPO, the techniques boosted math reasoning scores and improved agentic search tasks substantially across seven datasets. Small, principled tweaks to the anchor beat elaborate redesigns.
Catalog record
| CREATOR | Lunjun Zhang, Jimmy Ba |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 125 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-04417