What Does Flow Matching Bring To TD Learning?
Bhavya Agrawalla, Michal Nauman, Aviral Kumar · 2026
Read What Does Flow Matching Bring To TD Learning? by Bhavya Agrawalla, Michal Nauman, Aviral Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-04333.txt.
Introduction
Agrawalla and colleagues explain why flow matching aids temporal difference learning. The gain is not from modeling return distributions, which can even hurt results. Reading values by integration damps early errors at test time, unlike monolithic critics. Dense velocity supervision at many steps keeps features plastic for shifting targets. Flow critics thus beat monolithic critics by about twice on final results and five times on sample efficiency where plasticity fails.
Catalog record
| CREATOR | Bhavya Agrawalla, Michal Nauman, Aviral Kumar |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 97682 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.04333
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
What Does Flow Matching Bring To TD Learning? Bhavya Agrawalla, Michal Nauman, Aviral Kumar Abstract Abstract: Recent work shows that flow matching can be effective for scalar Q-value function estimation in reinforcement learning (RL), but it remains unclear why or how this approach differs from standard critics. Contrary to conventional belief, we show that their success is not explained by distributional RL, as explicitly modeling return distributions can reduce performance. Instead, we argue that the use of integration for reading out values and dense velocity supervision at each step of this integration process for training improves TD learning via two mechanisms. First, it enables robust value prediction through test-time recovery , whereby iterative computation through integration dampens errors in early value estimates as more integration steps are performed. This recovery mechanism is absent in monolithic critics. Second, supervising the velocity field at multiple interpolant values induces more plastic feature learning within the network, allowing critics to represent non-stationary TD targets without discarding previously learned features or overfitting to individual TD targets encountered during training. We formalize these effects and validate them empirically, showing that flow-matching critics substantially outperform monolithic critics (2 × \times in final performance and around 5 × \times in sample efficiency) in settings where loss of plasticity poses a challenge e.g., in high-UTD online RL problems, while remaining stable during learning. 1 Introduction Figure 1 : Flow-matching critics with non-stationary TD targets. Values are computed by integrating a learned velocity field over multiple steps. This iterative process enables test-time recovery (left), where errors in early integration steps are dampened by later steps. When TD targets change ( \theta\rightarrow\theta^{\prime} ), the integration dynamics can absorb part of the shift, allowing earlier features to remain largely unchanged, improving plasticity . Recent works show that utilizing flow-matching can be highly effective for value estimation in off-policy reinforcement learning (RL) [ 1 , 18 , 16 , 60 , 9 ] . These flow-matching critics depart from standard “monolithic” architectures that map state-action pairs to scalar Q-values in a single forward pass, by instead estimating values via the iterative integration of a learned velocity field given a noise input. This approach yields substantial empirical gains, and is more robust than monolithic networks in offline [ 31 ] and offline-to-online RL settings [ 39 ] . However, while the performance gap is clear, the mechanism driving it is unexplained. Do flow-matching critics succeed because they model value distributions, or because they introduce an inductive bias that mitigates some pathologies of TD learning? A natural hypothesis is that flow-matching critics succeed because they implicitly model return distributions, similar to distributional RL [ 6 ] . Indeed, much recent work applying flow matching to value learning [ 18 , 16 , 60 , 9 ] explicitly adopts a distributional RL perspective and derives objectives accordingly. We design controlled experiments to study the role of distributional RL [ 6 , 14 ] and find that it does not explain the observed gains: explicitly incorporating distributional updates often degrades performance relative to simple “expected-value” backups, which do not learn the return distribution. Flow-matching critics trained with standard TD learning (i.e., expected Q-value backups) also consistently outperform strong distributional RL algorithms. These results indicate that the advantage of flow matching does not come from modeling return distributions. So why do flow-matching critics work? We argue that flow-matching critics improve TD learning because they train a velocity field with dense supervision covering an integration trajectory used for inference (Figure 1 ). This form of iterative computation plays a dual role. First, at inference, the integration process enables the critic to refine its prediction as long as the velocity field is densely supervised to fit the target for multiple interpolant values and through the integration trajectory. We term this phenomenon test-time recovery , a phenomenon that arises because iterative computation on a densely supervised velocity field is used to estimate the Q-value. Second, beyond test-time recovery, flow-matching induces more plastic features that allow better fitting subsequent TD targets. Specifically, the velocity network’s features need not change substantially in attempts to fit successive TD targets, as these shifts can instead be accommodated (at least partially) through the integration process. This mitigates pathologies associated with plasticity loss [ 37 ] or representational capacity [ 29 ] . Put together, flow critics outperform monolithic architectures with identical computation graphs because of no dense velocity supervision. Theoretically , we formalize the notions of test-time recovery and the preservation of plasticity in simple settings, and show that flow matching induces weight update dynamics that preserve and reweight previously learned features rather than overwriting them under non-stationary TD targets. Such dynamics are absent in monolithic critics or their ensembles that do not use such iterative computation architectures. Achieving similar behavior in these baseline approaches instead requires auxiliary objectives or explicit regularization that bias optimization and must be carefully tuned. Empirically , we show that flow-matching critics tolerate substantially higher noise, are more robust to interventions such as resetting or freezing network components, and learn more isotropic features, thereby addressing several pathologies that have been reported with noise and regularization in TD targets. We also find that flow-matching critics achieve much stronger performance, resulting in a 2 \times performance gain and a 5 \times improvement in sample efficiency in high update-to-data online RL with offline data, a setting that is often plagued by loss of plasticity. Flow-matching critics also exhibit more stable learning at high UTD values. Finally, we show that these gains arise from fitting velocities rather than directly regressing to TD targets. Our main contribution is an understanding of why flow matching improves TD learning. We show that value estimation with an iterative integration procedure, and training the velocity field with dense supervision along the integration path, fundamentally alters both inference-time behavior and representation learning. In particular, this structure enables test-time recovery and promotes more plastic feature representations. Together, these effects mitigate the core pathologies associated with TD learning. 2 Related Work Flow matching in RL. Flow matching [ 2 , 33 ] and diffusion models [ 25 , 56 ] are used as policies in RL [ 51 , 8 , 38 , 34 , 59 , 49 ] , motivated primarily from the lens of expressivity [ 11 ] but also more recently through iterative computation [ 46 ] . Flow matching has been applied to value learning by parameterizing critic networks as velocity fields in TD learning [ 1 , 18 , 16 , 60 , 9 ] . While prior work shows strong results, it is unclear whether these gains stem from capacity, distributional modeling [ 18 , 16 , 60 , 9 ] , or the learning dynamics induced by iterative computation [ 1 ] . We provide evidence for the latter, showing that iterative computation from flow-matching produces more plastic features and improves value prediction. Stability and plasticity in TD learning. Prior work has identified several pathologies in TD learning, including value overestimation [ 23 , 22 ] , growth in parameter norms [ 44 , 40 ] , and rapid loss of plasticity [ 43 , 37 ] as training progresses. These issues are commonly attributed to