ResWM: Residual-Action World Model for Visual RL
Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim · 2026
Read ResWM: Residual-Action World Model for Visual RL by Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-11110.txt.
Introduction
Jseen Zhang and coauthors rethink world models for visual reinforcement learning. They predict the future from residual actions, small adjustments to the previous action, rather than absolute actions. Absolute action distributions are task-dependent and unknown in advance, which destabilizes optimization. Residual control matches the smoothness of real movement and shrinks the search space. Paired with an encoder that models changes between adjacent frames inside a Dreamer-style latent model, the approach improves sample efficiency, returns, and control smoothness on a standard control suite.
Catalog record
| CREATOR | Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44234 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.11110
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ResWM: Residual-Action World Model for Visual RL Jseen Zhang, Gabriel Adineera, Jinzhou Tan, Jinoh Kim Abstract . Learning predictive world models from raw visual observations is a central challenge in reinforcement learning (RL), especially for robotics and continuous control. Conventional model-based RL frameworks directly condition future predictions on absolute actions, which makes optimization unstable: the optimal action distributions are task-dependent, unknown a priori, and often lead to oscillatory or inefficient control. To address this, we introduce the Residual-Action World Model (ResWM), a new framework that reformulates the control variable from absolute actions to residual actions—incremental adjustments relative to the previous step. This design aligns with the inherent smoothness of real-world control, reduces the effective search space, and stabilizes long-horizon planning. To further strengthen the representation, we propose an Observation Difference Encoder that explicitly models the changes between adjacent frames, yielding compact latent dynamics that are naturally coupled with residual actions. ResWM is integrated into a Dreamer-style latent dynamics model with minimal modifications and no extra hyperparameters. Both imagination rollouts and policy optimization are conducted in the residual-action space, enabling smoother exploration, lower control variance, and more reliable planning. Empirical results on the DeepMind Control Suite demonstrate that ResWM achieves consistent improvements in sample efficiency, asymptotic returns, and control smoothness, significantly surpassing strong baselines such as Dreamer and TD-MPC. Beyond performance, ResWM produces more stable and energy-efficient action trajectories, a property critical for robotic systems deployed in real-world environments. These findings suggest that residual action modeling provides a simple yet powerful principle for bridging algorithmic advances in RL with the practical requirements of robotics. 1. Introduction Learning world models from high-dimensional visual observations stands as a cornerstone challenge in reinforcement learning (RL) ( 20 ; 24 ; 51 ; 79 ; 76 ; 74 ; 64 ; 69 ; 70 ) , demanding the orchestration of representation learning, dynamics prediction, and policy optimization in tandem ( 36 ; 49 ; 77 ; 65 ; 66 ; 78 ; 73 ) . In contrast to state-based paradigms where low-dimensional inputs transparently expose underlying system dynamics, visual RL grapples with multifaceted complexity, especially in robotics where agents navigate long-horizon tasks amid perpetually shifting environments ( 43 ; 71 ; 72 ) . Although model-free approaches such as SAC ( 60 ) and PPO ( 48 ) have garnered impressive achievements, their inherent sample inefficiency severely limits their viability in real-world scenarios, where data acquisition is costly, limited, or fraught with risks. Model-based reinforcement learning (MBRL) ( 43 ; 47 ) emerges as a promising antidote, cultivating an internal world model that facilitates efficient planning and policy refinement via imaginative simulations ( 44 ; 56 ) . Yet, the traditional manner of embedding actions within these models has surfaced as a pivotal bottleneck, hindering the full realization of MBRL’s potential. Predominant world models condition their latent dynamics directly on absolute actions , a seemingly intuitive choice that nonetheless instills suboptimal inductive biases with far-reaching implications ( 12 ; 80 ; 67 ; 75 ; 6 ) . Primarily, this setup casts policy learning as a high-variance conundrum, compounded by the task-specific, non-stationary distributions of optimal absolute actions. Furthermore, it frequently engenders oscillatory or erratic control trajectories, undermining planning efficacy and introducing safety hazards in physical embodiments. These shortcomings underscore a profound disconnect between the algorithmic scaffolding of world models and the imperatives of smooth, resilient control in embodied agents. In this paper, we champion the notion that the smoothness intrinsic to continuous control transcends mere desirability, evolving into a foundational principle ripe for exploitation. Our pioneering insight revolves around the residual change between successive actions—a quantity far more predictable and tractable than its absolute counterpart. Leveraging this, we unveil the Residual-Action World Model (ResWM) , a transformative framework that reparameterizes the control variable from absolute actions to residual actions . This reformulation embeds a robust temporal smoothness prior into the action space, fundamentally curtailing learning complexity by modeling incremental refinements rather than outright commands. In doing so, ResWM harmonizes with the continuum of physical dynamics, furnishing a steadfast bedrock for extended-horizon planning and fostering control signals that are inherently stable and energy-efficient. To anchor the control signal in the most salient perceptual insights, we complement our residual action paradigm with an innovative Observation Difference Encoder (ODL) . Diverging from the norm of independently encoding static frames, the ODL meticulously distills the embedded dynamics from the disparities between adjacent observations ( 54 ; 68 ; 63 ; 59 ; 35 ) . This yields a compact, dynamics-aware latent representation that seamlessly aligns with the prediction of residual actions, effectively sieving out static redundancies to spotlight temporal shifts causally pivotal to the agent’s adaptive adjustments. By prioritizing these differential cues, the ODL imbues the model with a sharpened focus on action-induced changes, enhancing its causal reasoning capabilities. ResWM is meticulously engineered for effortless assimilation into established Dreamer-style architectures, entailing only minimal alterations and eschewing any novel hyperparameters. Imagination-driven planning and policy optimization unfold wholly within the residual-action domain, transmuting exploration into fluid, localized perturbations rather than volatile, high-variance explorations. This paradigm shift not only bolsters learning stability and sample efficiency but also yields action trajectories that are demonstrably smoother and more energy-conserving—indispensable attributes for the secure integration of robotic systems in real-world contexts. We substantiate ResWM’s efficacy on the DeepMind Control Suite , a benchmark par excellence for continuous visual control. Empirical evaluations reveal ResWM’s consistent supremacy in sample efficiency, asymptotic performance, and control smoothness, markedly eclipsing formidable baselines such as Dreamer and TD-MPC. Complementary qualitative scrutiny discloses ResWM’s propensity for stabler latent dynamics and mitigated compounding prediction errors, affirming its robustness and versatility. In summary, this work advances the following contributions: • We introduce the Residual-Action World Model (ResWM) , a novel MBRL framework that reparameterizes the action space to instill a powerful smoothness prior, dramatically alleviating learning complexity and augmenting stability. • We propose the Observation Difference Encoder , an architecture that forges a dynamics-aware latent representation by explicitly capturing temporal changes in consecutive visual inputs. • We empirically validate that ResWM delivers substantial performance enhancements across intricate control tasks, providing a principled conduit to reconcile deep RL advancements with the exigencies of real-world robotics. 2. RELATED WORKS Model-based reinforcement learning (MBRL) tackles visual RL by learning world models from pixel observations, offering efficiency over model-free approaches like SAC and PPO. Dreamer ( 21 ; 27 ; 9 ; 38 ; 39 ; 41 ; 40 ) employs recurrent state-space models (RSSMs) for latent dynamics, while TD-MPC ( 29 ; 37 ) optimizes trajectories in latent space. Recent advances, such as DeepRAD ( 16 ) with robust representations and RAD ( 32 ) with data augmentation, condition dynamics on absolute actions. This approach introduces high-variance policies and oscillatory control, limiting stability in robotic tasks—where ResWM instead uses residual actions for smoother, more stable optimization. To address action instability, residual policies ( 52 ) predict incremental adjustments, as seen in incremental learning ( 46 ) , reducing variance in planning. Robotics methods like Gaussian processes ( 7 ) and filtering ( 11 ) enforce smoothness for energy efficiency. ResAct ( 2 ) applies residuals in actor-critic setups, improving stability on DMControl (e.g., 715 vs. 690