When Does Non-Uniform Replay Matter in Reinforcement Learning?

Michal Korniak, Mikołaj Czarnecki, Yarden As, Piotr Miłoś, Pieter Abbeel, Michal Nauman, [0.5em] ETH Zurich, [0.3em] Equal contribution · 2026

Read When Does Non-Uniform Replay Matter in Reinforcement Learning? by Michal Korniak, Mikołaj Czarnecki, Yarden As, Piotr Miłoś, Pieter Abbeel, Michal Nauman, [0.5em] ETH Zurich, [0.3em] Equal contribution (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10236.txt.

Description

Michal Korniak and five colleagues ask a pointed question about reinforcement learning. That is the field where agents learn by trial and reward. When does sampling past experience unevenly actually beat sampling it evenly? Their 2026 paper studies experience replay, the buffer of past transitions that off-policy algorithms reuse during training.

Across varied settings the authors find three factors govern the payoff. Replay volume means replays per environment step. Expected recency means how fresh sampled transitions are. Entropy means the spread of the sampling distribution. Uneven replay helps most when volume is low, and high entropy matters even at similar recency. Their Truncated Geometric sampler favors recent experience while staying spread out. It lifted sample efficiency in low-volume regimes and stayed competitive when volume was high.

Catalog record

CREATORMichal Korniak, Mikołaj Czarnecki, Yarden As, Piotr Miłoś, Pieter Abbeel, Michal Nauman, [0.5em] ETH Zurich, [0.3em] Equal contribution
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10236

Browse the full library · About the Tower of Babel