Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow
Yunyue Wei, Chenhui Zuo, Yanan Sui · 2026
Read Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow by Yunyue Wei, Chenhui Zuo, Yanan Sui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19707.txt.
Introduction
Yunyue Wei, Chenhui Zuo, and Yanan Sui work on controlling systems with many moving parts. Robots and biological systems can have huge state and action spaces. Exploring those spaces well is essential, yet standard exploration in reinforcement learning is undirected. Its quality falls sharply as the action space grows.
Many methods shrink the space first, but that limits what the policy can express. The authors propose exploration guided by value estimates, steered through a flow-based method. Guidance should focus trials where they teach the most. Keeping the full action space preserves the system's flexibility. The work targets high-dimensional control in robotics and biology, where random flailing learns almost nothing.
Catalog record
| CREATOR | Yunyue Wei, Chenhui Zuo, Yanan Sui |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-19707
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Scalable Exploration for High-Dimensional Continuous Control via Value-Guided Flow Yunyue Wei, Chenhui Zuo, Yanan Sui Abstract Controlling high-dimensional systems in biological and robotic applications is challenging due to expansive state–action spaces, where effective exploration is critical. Commonly used exploration strategies in reinforcement learning are largely undirected with sharp degradation as action dimensionality grows. Many existing methods resort to dimensionality reduction, which constrains policy expressiveness and forfeits system flexibility. We introduce Q-guided Flow Exploration ( Qflex ), a scalable reinforcement learning method that conducts exploration directly in the native high-dimensional action space. During training, Qflex traverses actions from a learnable source distribution along a probability flow induced by the learned value function, aligning exploration with task-relevant gradients rather than isotropic noise. Our proposed method substantially outperforms representative online reinforcement learning baselines across diverse high-dimensional continuous-control benchmarks. Qflex also successfully controls a full-body human musculoskeletal model to perform agile, complex movements, demonstrating superior scalability and sample efficiency in very high-dimensional settings. Our results indicate that value-guided flows offer a principled and practical route to exploration at scale. 1 Introduction Controlling over high-dimensional dynamical systems underpins a broad range of applications in robotics, sports, and embodied intelligence from legged locomotion to full-body musculoskeletal control. Complex sensorimotor coordination and over-actuation are common in such systems. As the number of sensors and actuators grows, these systems gain flexibility and robustness, enabling agile and precise movements. However, increasing dimensionality also amplifies the challenges of coordination and learning efficiency due to rapidly expanding state–action spaces, making effective exploration strategies essential. A widespread practice in online deep reinforcement learning (RL) is to inject undirected stochasticity (e.g., Gaussian noise) into policy outputs for exploration ( 17 ) . While simple and effective in moderate dimensions, such isotropic perturbations rapidly lose coverage and become sample-inefficient as action dimensionality and actuator redundancy grow, yielding vanishing signal for discovering task-relevant actions. Dimensionality reduction-based learning constrains control within low-rank subspaces to make search tractable ( 1 ) . Such complementary strategy may forfeit the flexibility and redundancy that high-dimensional dynamical systems are designed to provide. Iterated sampling techniques, most notably diffusion models and flow-based transports, have achieved striking success in high-dimensional generative modeling, providing robust procedures for sampling in thousands of dimensions ( 42 ; 28 ) . Motivated by these advances, several works have adapted iterated-sampling ideas to control and decision making ( 22 ; 48 ) . Despite promising results in moderate-dimensional settings, these methods have not demonstrated success for high-dimensional continuous control with substantial over-actuation. In this paper, we introduce Q-guided Flow Exploration ( Qflex ), a scalable exploration mechanism that operates in the native high-dimensional action space. Qflex achieves directed exploration for policy improvement by sampling from probability flow induced by learned state–action value function Q , proposing informative actions aligned with task-relevant direction. Our design preserves flexibility of the complex systems, and integrates cleanly into a actor-critic loop, yielding efficient learning across diverse high-dimensional continuous-control benchmarks. Qflex also succeeds in controlling a full-body musculoskeletal system to perform complex movements, highlighting its scalable and efficient exploration. Our contributions : (1) We propose Qflex , a scalable RL method which achieves value-aligned directed exploration with policy-improvement validity, enabling efficient learning over high-dimensional state-action spaces. (2) We present an actor–critic implementation of Qflex that consistently outperforms representative Gaussian-based and diffusion-based RL baselines on a wide range of high-dimensional continuous-control benchmarks. (3) We demonstrate Qflex on a full-body human musculoskeletal model with 700 actuators, achieving agile, complex movements and efficient exploration without dimension reduction. 2 Related Work High-dimensional over-actuated control. The control of high-dimensional dynamical system is challenging due to its high-dimensionality and over-actuation. With few model-based strategies ( 18 ; 47 ) , model-free deep reinforcement learning is the mainstream solution for solving complex control tasks ( 23 ; 15 ; 5 ) . Hierarchical RL decomposes decision making into high-level planning and low-level control, reducing exploration burden by restricting search to joint- or skill-level choices ( 26 ; 34 ; 14 ) . Curriculum-based learning iterates over sub-tasks to smooth the learning curve for diverse skill learning over high-dimensional embodiment ( 4 ; 33 ) . DEP-RL employs bio-inspired sampling for coordinated exploration ( 39 ; 38 ) . Lattice generated correlated noise for exploration by injecting stochasticity into latent embeddings of the policy network ( 11 ; 41 ) . Synergy-based approaches such as DynSyn ( 19 ) learn or impose low-dimensional control subspaces derived from morphology or task structure, enabling more stable training on systems with high degrees of freedom. These methods primarily mitigate undirected exploration issue by explicit or implicit dimensionality reduction, which can constrain policy expressiveness and underutilize redundancy—potentially limiting the flexibility required for agile, task-diverse movements. Iterated sampling-based online reinforcement learning. Inspired by early successes of iterated sampling for offline and imitation decision making ( 22 ; 9 ) , many works have adapted diffusion-based policy parameterization to online reinforcement learning to encourage diverse action distribution ( 48 ; 27 ; 20 ; 7 ) . DACER introduces a diffusion actor–critic with an entropy regulator to stabilize policy learning and maintain exploration ( 46 ) . Given unknown target distributions, several studies utilize the learned value function to regularize the policy learning ( 12 ; 13 ; 21 ) . QSM matches the score of diffusion policy with the gradient of the Q-function ( 35 ) . SDAC introduces a Q-reweighted score matching function to avoid unstable training of backpropagating gradients through the diffusion chain ( 31 ) . Recent works also employ flow-based policy in KL-constrained policy optimization ( 30 ; 32 ) . These methods typically use standard Gaussian as a general initial distribution for the primary goal of multi-modal policy learning. The uninformative, isotropic bases can hinder scalability of policy learning in high-dimensional continuous control. 3 Preliminaries 3.1 High-dimensional continuous control In this paper, we formalize the control of high-dimensional dynamical system as a infinite horizon Markov decision process (MDP) defined by the tuple \mathcal{M}\coloneqq\{\mathcal{S},\mathcal{A},\gamma,f,r,\rho\} , where \mathcal{S}\subset\mathbb{R}^{|\mathcal{S}|} is the state space, \mathcal{A}\subset[-1,1]^{|\mathcal{A}|} is the action space, \gamma is the discount factor, f\coloneqq\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{P}(\mathcal{S}) is the transition probability to \bm{s}^{\prime}\in\mathcal{S} when being in \bm{s}\in\mathcal{S} and executing \bm{a}\in\mathcal{A} , r\coloneqq\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function, and \rho\coloneqq\mathcal{S}\rightarrow\mathcal{P}(\mathcal{S}) is the initial state distribution. The MDP starts from an initial state \bm{s}_{0} sampled from \rho , and proceeds with actions sampled from a policy \pi\coloneqq\mathcal{S}\rightarrow\mathcal{P}(\mathcal{A}) . Our goal is to optimize the policy parameters to maximize the discounted cumulative reward: \displaystyle J(\pi)=\mathbb{E}_{\pi}\sum_{h=0}^{\infty}\gamma^{h}r(\bm{s}_{h},\bm{a}_{h}). (1) Compared with low-dimensional dynamical systems, controlling high-dimensional dynamics is substantially more challenging. We use the human musculoskeletal system as a motivating example to illustrate the difficulties of high-dimensional continuous control. High dimensionality. Full-body human locomotion integrates rich sensory feedback with more than 600 muscles. Unlike robotic arms or quadrupeds, which typically operate within state and action spaces on the order of tens, the human musculoskeletal system features state and action spaces that are orders of magnitude larger. The size of the state-action space grows rapidly with dimension, leading to pronounced “curse-of-dimensionality” effects ( 24 ) . This demands expressive models and substantial informative data to reliably map high-dimensional states and actions to control performance. Over-actuation. The number of biological actuators far exceeds the system’s