Reduce Computational Cost In Deep Reinforcement Learning Via Randomized Policy Learning
Zhuochen Liu, Rahul Jain, Quan Nguyen · 2025
Read Reduce Computational Cost In Deep Reinforcement Learning Via Randomized Policy Learning by Zhuochen Liu, Rahul Jain, Quan Nguyen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19054.txt.
Introduction
Zhuochen Liu, Rahul Jain, and Quan Nguyen cut training cost for robot leg control in a 2025 arXiv work whose head gives the title RANDPOL. Deep nets for leg control hold vast counts of weights that all shift in training. RANDPOL freezes the inner layers at random starts and trains only the last straight layer. The frozen parts still bend signals in rich ways.
Tests used a Unitree Go2 robot dog in sim form and on real floors. RANDPOL matched a known strong method on walk skill with far fewer trained weights. Each training step also took less time. The learned control moved from sim to the real robot with no fresh training.
Catalog record
| CREATOR | Zhuochen Liu, Rahul Jain, Quan Nguyen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19054
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Reduce Computational Cost In Deep Reinforcement Learning Via Randomized Policy Learning Zhuochen Liu, Rahul Jain, Quan Nguyen Abstract Recent advancements in reinforcement learning (RL) have leveraged neural networks to achieve state-of-the-art performance across various control tasks. However, these successes often come at the cost of significant computational resources, as training deep neural networks requires substantial time and data. In this paper, we introduce an actor-critic algorithm that utilizes randomized neural networks to drastically reduce computational costs while maintaining strong performance. Despite its simple architecture, our method effectively solves a range of control problems, including the locomotion control of a highly dynamic 12-motor quadruped robot, and achieves results comparable to leading algorithms such as Proximal Policy Optimization (PPO). Notably, our approach does not outperform other algorithms in terms of sample efficnency but rather in terms of wall-clock training time. That is, although our algorithm requires more timesteps to converge to an optimal policy, the actual time required for training turns out to be lower. 1 Introduction Control problems are often framed as sequential decision-making tasks and modeled using Markov Decision Processes (MDPs). These problems are typically solved by learning an optimal control policy through Reinforcement Learning (RL). When the state and action spaces are finite and the system model is known, optimal policies can be derived using dynamic programming (DP) algorithms. However, in many real-world applications, such as playing the board game Go or controlling a quadruped robot, the state and action spaces are often large or continuous, and the system model is unknown, making traditional DP algorithms inapplicable. To address the challenge of unknown system models, model-free RL algorithms ( Mnih et al. 2015 ; Schulman et al. 2015a ; Mnih et al. 2016 ; Haarnoja et al. 2018 ; Lillicrap et al. 2015 ) rely on empirical sample-based approximations rather than explicit models. While these methods require a data generator, this requirement is often not restrictive. For example, game-playing agents can explore strategies by playing the game, and locomotion controllers can learn policies using high-fidelity physics simulations. Numerous studies have demonstrated the effectiveness of model-free RL across various domains ( Silver et al. 2016 ; Silver et al. 2018 ; Bellegarda et al. 2022 ; Bellegarda & Ijspeert 2022 ; Kumar et al. 2021 ; Siekmann et al. 2021 ). To handle continuous state and action spaces, approximation-based RL algorithms ( Kakade 2001 ; Van Hasselt & Wiering 2007 ; Wang et al. 2019 ) have been proposed. These methods approximate the value function and policy to generalize across inputs and outputs. In recent years, neural networks have become the dominant choice for function approximation due to their scalability, allowing them to handle increasingly complex problems ( Kendall et al. 2019 ; Shalev-Shwartz et al. 2016 ; Raghu et al. 2017 ; Liu et al. 2017 ). Modern RL algorithms are typically categorized as policy-based ( Sutton et al. 1999 ; Gu et al. 2016 ), value-based ( Watkins & Dayan 1992 ; Mnih et al. 2013 ), or actor-critic ( Konda & Tsitsiklis 1999 ; Wu et al. 2017 ), which combines benefits of both approaches. Nevertheless, model-free algorithms are slow to converge because they require a large number of samples to achieve a good estimation of the expectation. The application of deep neural networks amplifies this problem due to their high amount of trainable parameters and the consequent need of sufficient training data to avoid overfitting. In this paper, we focus on continuous MDP problems, where both the state and action spaces are continuous, hence requiring function approximation for both the policy and value functions. Specifically, our work adopts randomized neural networks as function approximators, where only the last layer is fine-tuned while all other hidden layers remain randomly initialized. Previous study ( Rahimi & Recht 2008a ; Rahimi & Recht 2008b ; Gallicchio & Scardapane 2020 ) has shown that such networks can match or even surpass the performance of fully trainable neural networks while significantly reducing the number of trainable parameters. To leverage the advantages of randomized neural networks, we propose the RANDomized POlicy Learning (RANDPOL) algorithm. RANDPOL is an actor-critic algorithm that employs randomized neural networks for both the policy and value functions. The actor network is updated via stochastic policy gradient and the critic network is updated via temporal difference. Fine-tuning the last layer can be interpreted as learning a linear combination of random basis functions from a function space. Notably, RANDPOL not only matches the performance of fully trainable neural networks but also requires less time for training. We conduct comprehensive evaluations in OpenAI Gym ( Brockman et al. 2016 ) environments and compare our approach with other widely used stochastic on-policy algorithms such as Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ), Advantage Actor Critic (A2C) ( Mnih et al. 2016 ) , and Trust Region Policy Optimization (TRPO) ( Schulman et al. 2015a ) . Additionally, we demonstrate RANDPOL’s real-world applicability by learning an end-to-end locomotion control policy for a quadruped robot. 2 Preliminaries Random function approximation error bound analysis. Because our algorithm is based on randomized neural networks, it is important to know how well the randomized function basis is able to approximate other classes of functions that we are interested in. Rahimi & Recht 2008a shows that a target function within a Reproducing Kernel Hilbert Space (RKHS) can be approximated by certain kind of nonlinear random functions with an error within O(1/\sqrt{K}) with high probability, where K is the number of random functions. Rahimi & Recht 2008b further expands the conclusion that random shallow neural networks can be used as random function bases for approximation with a bounded error of O(1/\sqrt{K}) . In addition, Gallicchio & Scardapane 2020 shows that deep random neural networks can also be used as random function bases to reduce the training time in magnitudes, with a trade off of very small or even negligible loss of accuracy. Empirical bellman operator. In reinforcement learning, an optimal value function is learned via a Bellman Operator G:\mathcal{F}(\mathcal{X})\rightarrow\mathcal{F}(\mathcal{X}) such that [Gv](x)=\min_{u\in\mathcal{U}}\{r(x,u)+\gamma\mathbb{E}[v(x^{\prime})|x,u]\}\forall x\in\mathcal{X} . However, when the system model is unknown, computing the expectation becomes challenging. Haskell et al. 2019 shows that we can replace the Bellman Operator G with an empirical Bellman Operator \widehat{G} such that [\widehat{G}v](x)=\min_{u\in\mathcal{U}}\{r(x,u)+\gamma\frac{\gamma}{n}\sum_{i=1}^{n}v(x_{i}^{\prime})\} , where \{x_{i}^{\prime}\}_{i=1}^{n} are samples of the next state collected from a generator given the current state x and the performed action u . Furthermore, Haskell et al. 2019 shows that by combining the empirical Bellman Operator \widehat{G} with function approximation and applying \widehat{G} to v sufficiently many iterations K , v_{K} will converge to the fixed point v^{*} with a high probability. Problem formulation. Consider an infinite horizon discounted reward continuous MDP \mathcal{M}=(\mathcal{X},\mathcal{U},P,r,\gamma) where \mathcal{X} is the state space, \mathcal{U} is the action spaces, P(\cdot|x,u) is the transition probability kernel, r:\mathcal{X}\times\mathcal{U}\rightarrow\mathbb{R} is the reward function, and \gamma\in(0,1) is the discount factor. Let \Pi denote the class of stochastic markov policies mappings \pi:\mathcal{X}\rightarrow\mathcal{P}({\mathcal{U}}) , the space of probability distributions over the action space. For a trajectory \tau=(x_{0},\pi(x_{0}),x_{1},\pi(x_{1}),\dots,x_{T},\pi(x_{T})) of length T , we represent its discounted return as R(\tau)=\sum_{t=0}^{T}\gamma^{t}r_{t} and its probability given policy \pi as P(\tau|\pi)=\rho_{0}(x_{0})\prod_{t=0}^{T-1}P(x_{t+1}|x_{t},u_{t})\pi(u_{t}|x_{t}) , where \rho_{0} is the probability distribution of the initial state of