Stackelberg Coupling of Online Representation Learning and Reinforcement Learning
Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen · 2025
Read Stackelberg Coupling of Online Representation Learning and Reinforcement Learning by Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-07452.txt.
Introduction
Deep Q-learning trains one network to both see the world and judge actions, and that coupling can destabilize learning. Fernando Martinez and co-authors propose SCORER, a framework that splits the two jobs into a leader and a follower, as in a Stackelberg game. The value network leads, updating slowly and committing to its strategy.
The perception network follows, adapting faster to produce representations that cut the variance of the value target. This division turns learning into a two-level optimization run on two timescales. In experiments, the setup reduced bias in value estimates and kept co-adaptation stable where single-network designs wobble. The authors present the leader-follower split as a general fix for a known weakness of value-based reinforcement learning.
Catalog record
| CREATOR | Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 73 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-07452
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Stackelberg Coupling of Online Representation Learning and Reinforcement Learning Fernando Martinez, Tao Li, Yingdong Lu, Juntao Chen Abstract Integrated, end-to-end learning of representations and policies remains a cornerstone of deep reinforcement learning (RL). However, to address the challenge of learning effective features from a sparse reward signal, recent trends have shifted towards adding complex auxiliary objectives or fully decoupling the two processes, often at the cost of increased design complexity. This work proposes an alternative to both decoupling and naive end-to-end learning, arguing that performance can be significantly improved by structuring the interaction between distinct perception and control networks with a principled, game-theoretic dynamic. We formalize this dynamic by introducing the Stackelberg Coupled Representation and Reinforcement Learning (SCORER) framework, which models the interaction between perception and control as a Stackelberg game. The perception network (leader) strategically learns features to benefit the control network (follower), whose own objective is to minimize its Bellman error. We approximate the game’s equilibrium with a practical two-timescale algorithm. Applied to standard DQN variants on benchmark tasks, SCORER improves sample efficiency and final performance. Our results show that performance gains can be achieved through principled algorithmic design of the perception-control dynamic, without requiring complex auxiliary objectives or architectures. 1 Introduction The introduction of deep reinforcement learning (RL) marked a significant milestone, exemplified by algorithms capable of achieving human-level control in complex tasks like Atari games directly from pixel inputs mnih2015human . This success popularized the end-to-end paradigm, where representation learning and policy optimization are performed jointly within a single deep neural network, driven primarily by the RL objective function. Building on these initial successes, the integrated learning approach rapidly evolved, enabling deep RL systems to solve intricate strategy games vinyals2019grandmaster ; berner2019dota , mastered complex robotic manipulation tasks schulman2017proximalpolicyoptimizationalgorithms ; ze2023h , and showed promise in areas such as autonomous driving and resource management kiran2021deep . The main appeal of this paradigm derives from its conceptual simplicity and the potential for features and policies to co-adapt, promising benefits such as stabilized learning and improved sample efficiency. However, relying solely on the often sparse or delayed reward signal to shape internal representations can be challenging pmlr-v139-stooke21a , which further hinders RL performance within the current task. Recognizing this, a significant body of work has focused on improving representation learning within the deep RL framework. One prominent direction involves augmenting the primary RL objective with auxiliary losses derived from supervised, unsupervised, and self-supervised learning. These tasks provide additional learning signals, incentivizing the network to capture structural information about the environment, such as predicting future states, reconstructing observations, or ensuring temporal consistency jaderberg2017reinforcement ; schwarzerdata ; pmlr-v119-laskin20a . These methods often retain the end-to-end structure but enrich the gradient signal for the shared encoder network. More recently, inspired by the success of self-supervised learning in computer vision grill2020bootstrap ; chen2020simple , a particular trend has emerged exploring the separation of representation learning from the direct reward-driven reinforcement learning objective. Some approaches explicitly separate these processes, aiming to learn powerful representations independently using self-supervised learning (e.g., contrastive learning pmlr-v139-stooke21a ) that may run concurrently with, but are distinct from, the RL policy learner. While methods focusing on enhanced or decoupled representation learning have demonstrated strong empirical results pmlr-v139-stooke21a ; fujimoto2023sale , they often introduce significant architectural or training complexities. These can include the need for carefully designed auxiliary tasks, management of multiple interacting loss functions, or separate pre-training phases, potentially increasing manual design effort and overall computational demands. Furthermore, decoupling might forgo the potential benefits of tight, dynamic co-adaptation between representation and policy, specifically tailored for the control task at hand. This work revisits the integrated learning paradigm but proposes a more structured and principled interaction between representation learning and reinforcement learning. We posit that the limitations of naive end-to-end learning can be addressed not only by adding external signals or by complete decoupling, but also by fundamentally rethinking the internal dynamics of how representations and policies co-evolve (illustrated in Figure 1 ). If the representation learner can strategically anticipate the needs of the policy learner, it might lead to more effective and efficient overall learning without the overhead of fully separate systems. Figure 1: Overview of the SCORER agent and its internal dynamics. (Left) Overall agent-environment interaction loop. Internally, the agent comprises a perception network (Leader, f_{\phi} ) and a control network (Follower, Q_{\theta} ) that interact via Stackelberg game dynamics ( U_{L},U_{F} representing their utility/loss functions). The perception network produces features z=f_{\phi}(s) used by the control network. (Right) Details the Stackelberg interaction within the agent. To this end, we introduce the Stackelberg Coupled Representation and Reinforcement Learning (SCORER) framework. Drawing inspiration from Stackelberg games in economics and game theory Lambertini_2018 , SCORER models the interaction between representation learning (the Perception network) and reinforcement learning (the Control network) as a hierarchical game between two strategic agents with aligned but distinct goals. We designate the perception process as the leader. Its objective is to learn features that are maximally beneficial for the control process. The leader makes its decision (i.e., updates its parameters) anticipating the optimal response of the follower—the control process—which aims to minimize its own objective (e.g., Mean Squared Bellman Error) given the current representation. The integrated learning process is characterized by the associated Stackelberg equilibrium, for which we devise a two-timescale gradient descent algorithm to approximate (Section 4.2 ). Our code is open-sourced at https://anonymous.4open.science/r/SCORER-530 . In summary, our contributions are three-fold : 1) We introduce the SCORER framework , a new game-theoretic formulation for integrated perception and control learning in RL. 2) We develop a practical two-timescale algorithm to approximate the Stackelberg equilibrium, enabling anticipatory representation learning 3) Through extensive experiments , we demonstrate SCORER’s power to improve sample efficiency and performance of baseline value-based RL algorithms on benchmark tasks. 2 Related Work Representation learning has been a pivotal topic in RL, even before the advent of deep neural networks. Early stage efforts concentrated on value function approximation thrun1993issues ; Tsitsiklis97TD and associated basis function selections taomultiRL ; geramifard13linearapprox . Entering the age of deep learning, RL harnesses the representation power of neural networks and becomes capable of solving high-dimensional complex tasks with multi-modal inputs, such as texts deeprl-dialogue , images mnih2015human , and multi-modal sensor data liu17sensorfusion ; yin-tao24pirl ; tao25dt-pirl . The confluence of deep representation learning and RL leads to the vibrant research field of deep RL. Depending on the purpose, representation learning methods in deep RL can be classified into two major categories: those aimed at facilitating training and those designed to enhance testing performance. In the training phase, learning appropriate representations helps with 1) dimension reduction : extracting low-dimensional features from high-dimensional inputs, making the RL tasks tractable, a typical example of which is vision-based control tasks lesort18staterepcontrol ; tao23cola ; 2) sample efficiency : capturing underlying environment dynamics mahajan22approxinfo ; lee20predictive-rep , which reduce the number of training episodes and foster faster convergence; and 3) oriented exploration : directing future exploration strategies based on past experiences and intrinsic motivation kulkarni16intrinsic , particularly in reward-free and sparse-reward settings pathak17curiosity ; hazan19entropy-exp . Some other considerations include training stability greydanus2018visualizing and explainability dazeley23xrl ; tao23cola . Regarding testing improvement, previous work focuses