floq : Training Critics via Flow-Matching for Scaling Compute in Value-Based RL
Bhavya Agrawalla, Michal Nauman, Khush Agarwal, Aviral Kumar · 2025
Read floq : Training Critics via Flow-Matching for Scaling Compute in Value-Based RL by Bhavya Agrawalla, Michal Nauman, Khush Agarwal, Aviral Kumar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06863.txt.
Introduction
Value functions with adjustable internal computation are trained by flow matching in a 2025 reinforcement learning paper by Bhavya Agrawalla and three co-authors. Modern machine learning gets dense supervision at intermediate steps, as in next-token training or diffusion denoising. Value learning in reinforcement learning, by contrast, usually represents values in one monolithic block.
Their approach, floq, parameterizes the Q-function through a velocity field borrowed from flow-matching generative models. A temporal-difference objective trains that field, bootstrapping from a target field integrated over several steps. Turning the number of integration steps up or down scales the value function's capacity in a controlled way. Across hard offline reinforcement learning benchmarks and online fine-tuning tasks, performance improved by nearly twofold, and capacity scaled far better than with standard architectures.
Catalog record
| CREATOR | Bhavya Agrawalla, Michal Nauman, Khush Agarwal, Aviral Kumar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-06863
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
floq : Training Critics via Flow-Matching for Scaling Compute in Value-Based RL Bhavya Agrawalla, Michal Nauman, Khush Agarwal, Aviral Kumar Abstract Abstract: A hallmark of modern large-scale machine learning techniques is the use of training objectives that provide dense supervision to intermediate computations, such as teacher forcing the next token in language models or denoising step-by-step in diffusion models. This enables models to learn complex functions in a generalizable manner. Motivated by this observation, we investigate the benefits of iterative computation for temporal difference (TD) methods in reinforcement learning (RL). Typically they represent value functions in a monolithic fashion, without iterative compute. We introduce floq ( flow-matching Q-functions ), an approach that parameterizes the Q-function using a velocity field and trains it using techniques from flow-matching, typically used in generative modeling. This velocity field underneath the flow is trained using a TD-learning objective, which bootstraps from values produced by a target velocity field, computed by running multiple steps of numerical integration. Crucially, floq allows for more fine-grained control and scaling of the Q-function capacity than monolithic architectures, by appropriately setting the number of integration steps. Across a suite of challenging offline RL benchmarks and online fine-tuning tasks, floq improves performance by nearly 1.8 × \times . floq scales capacity far better than standard TD-learning architectures, highlighting the potential of iterative computation for value learning. 1 Introduction Figure 1 : floq architecture. Our approach models the Q-function via a velocity field in a flow-matching generative model. Over multiple calls, this velocity field converts a randomly sampled input \mathbf{z}(0) into a sample from the Dirac-delta distribution centered at the Q-value. We prescribe how this sample can be trained via a flow-matching loss. Doing this enables us to scale computation by running numerical integration, with multiple calls to the velocity field. To train floq , we utilize a categorical representation of input \mathbf{z}_{t} [ 13 ] . A key principle that has enabled effective model scaling in various areas of machine learning is the use of iterative computation : producing complex output functions by composing a sequence of small and simple operations. For example, transformers [ 72 ] are able to generate coherent text and long reasoning chains by predicting the next token and by composing multiple atomic reasoning strategies [ 19 ] respectively. Similarly, diffusion models [ 29 , 68 ] and flow-matching techniques [ 47 , 2 ] learn to synthesize realistic images by progressively denoising small perturbations. The wide application of these models suggests that iterative computation is a powerful tool for modeling complex functions with deep networks. Motivated by these empirical results, in this paper, we ask: can iterative computation also improve value estimation in reinforcement learning (RL)? Specifically, we are interested in improving the estimation of the Q-value function. While Q-functions map state-action inputs to a scalar value, they are known to be highly complex and difficult to fit accurately (e.g., [ 10 ] ). Standard temporal-difference (TD) learning used to train Q-functions struggles to leverage capacity of deep networks [ 38 , 39 , 5 , 50 , 21 ] , often resulting in poor generalization. These problems are further exacerbated in the offline RL problem setting [ 45 , 36 ] , where we must learn entirely from static datasets. This motivates exploring architectures that spend compute iteratively to estimate value functions, potentially yielding more accurate Q-values, and as a result, better downstream policies. A natural starting point to use iterative computation for value-based RL is to represent the Q-function with ResNets [ 27 ] , where stacking residual blocks iteratively computes the Q-value. Recent work in TD-learning has obtained modest gains with ResNets [ 40 , 41 , 13 , 53 ] , but these methods deliberately need to operate with normalization and regularizers to enable stable training [ 5 , 53 , 42 , 43 , 40 ] . Despite improvements, these approaches lack one ingredient that makes iterative computation effective in transformers or diffusion models: supervision at every step of the iterative computation process. Just as next-token prediction supervises each generated token, and diffusion supervises each denoising step, we hypothesize that stepwise loss supervision applied to TD learning might lead to improvements. With this observation, to effectively leverage iterative computation with dense supervision, we design a novel architecture for representing Q-functions. Instead of using a single monolithic network, we represent the Q-function as a velocity field over a scalar value (Figure 1 ). Our approach, floq (flow-matching Q-functions) samples a scalar uniformly distributed noise variable and maps it to the Q-value by numerically integrating the predictions of this velocity field. We train this velocity field with a linear flow-matching objective [ 2 , 47 ] adapted from generative modeling. Unlike standard flow-matching, which supervises the velocity with stationary targets, we train the velocity to match the evolving TD-targets. At each step, we minimize the deviation between the current Q-value estimate and the corresponding TD-target. We introduce several design choices that stabilize training and help the architecture scale capacity effectively. First, we appropriately set the support of the initial noise to be as wide as possible. Second, we use a categorical input representation to handle the non-stationarity of inputs during training. Finally, we apply a tailored Fourier-basis embedding of time so that the velocity predictions vary meaningfully across integration steps. These design choices are crucial for learning a good floq critic. We use floq to represent the Q-function for a number of complex RL [ 45 , 36 ] tasks from the OGBench [ 57 ] benchmark, previously studied by Park et al. 2025c . In aggregate, we find that floq outperforms offline RL algorithms that represent Q-functions using a monolithic network by nearly 1.8\times . floq is superior even when these approaches are provided with more parameters, and more complex and higher capacity architectures. floq also outperforms existing methods when running online fine-tuning after offline RL pre-training. We also show that increasing the number of flow-matching steps results in better downstream policy performance. Allocating the same capacity via Q-network ensembles or ResNets performs worse, even in a compute-matched evaluation. Finally, our approach enables a pathway for “test-time scaling” [ 67 ] by increasing the number of integration steps to calculate Q functions. 2 Related Work Expressive generative policies in RL. A growing body of work in offline RL has emphasized the importance of moving beyond unimodal Gaussian policies [ 36 , 18 , 37 , 17 ] toward richer, generative function classes such as diffusion models [ 75 , 26 , 81 , 4 , 46 , 63 ] , flow-based policies [ 47 , 2 , 59 ] , and sequence models [ 32 , 44 , 80 ] . This shift is motivated by evidence that policy learning is often a significant bottleneck in offline RL [ 35 , 56 ] . Diffusion policies have been among the first expressive alternatives, but early variants such as Diffusion-QL [ 75 ] faced optimization challenges. Subsequent work has explored more stable formulations through implicit action re-ranking [ 26 ] , action-space gradient estimators [ 81 , 46 ] , or diffusion policy gradient methods [ 63 ] . Flow-matching policies have recently emerged