Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer Dr. Raghuram Bharadwaj is supported by the Anusandhan National Research Foundation (ANRF) under the Prime Minister Early Career Research Grant ANRF/ECRG/2024/005235/ENS

Sooraj Sathish, Keshav Goyal, Raghuram Bharadwaj Diddigi · 2025

Read Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer Dr. Raghuram Bharadwaj is supported by the Anusandhan National Research Foundation (ANRF) under the Prime Minister Early Career Research Grant ANRF/ECRG/2024/005235/ENS by Sooraj Sathish, Keshav Goyal, Raghuram Bharadwaj Diddigi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24947.txt.

Introduction

Sooraj Sathish, Keshav Goyal, and Raghuram Bharadwaj Diddigi wrote this machine learning paper, published in 2025. It asks how the inner features a deep reinforcement learning agent learns can be reused, through simple linear models, on new tasks.

Deep agents are costly to train, so reusing their learned features is attractive. The final hidden layer's activity can serve as a tidy summary of each game state. The authors find a snag: those features tend to move together, which blunts simple linear models built on them. They add a penalty during training that pushes the features apart. With cleaner features, a fresh linear model on a new task learns better. The work supports a cheap transfer path: train deep once, then adapt with light models.

Catalog record

CREATORSooraj Sathish, Keshav Goyal, Raghuram Bharadwaj Diddigi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-24947

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning Distinguishable Representations in Deep Q-Networks for Linear Transfer Dr. Raghuram Bharadwaj is supported by the Anusandhan National Research Foundation (ANRF) under the Prime Minister Early Career Research Grant ANRF/ECRG/2024/005235/ENS Sooraj Sathish, Keshav Goyal, Raghuram Bharadwaj Diddigi Abstract Deep Reinforcement Learning (RL) has demonstrated success in solving complex sequential decision-making problems by integrating neural networks with the RL framework. However, training deep RL models poses several challenges, such as the need for extensive hyperparameter tuning and high computational costs. Transfer learning has emerged as a promising strategy to address these challenges by enabling the reuse of knowledge from previously learned tasks for new, related tasks. This avoids the need for retraining models entirely from scratch. A commonly used approach for transfer learning in RL is to leverage the internal representations learned by the neural network during training. Specifically, the activations from the last hidden layer can be viewed as refined state representations that encapsulate the essential features of the input. In this work, we investigate whether these representations can be used as input for training simpler models, such as linear function approximators, on new tasks. We observe that the representations learned by standard deep RL models can be highly correlated, which limits their effectiveness when used with linear function approximation. To mitigate this problem, we propose a novel deep Q-learning approach that introduces a regularization term to reduce positive correlations between feature representation of states. By leveraging these reduced correlated features, we enable more effective use of linear function approximation in transfer learning. Through experiments and ablation studies on standard RL benchmarks and MinAtar games, we demonstrate the efficacy of our approach in improving transfer learning performance and thereby reducing computational overhead. I Introduction Reinforcement Learning (RL) [ 1 , 27 ] is a popular paradigm for solving sequential decision-making problems under uncertainty. When combined with deep neural networks, RL has achieved success in various domains, such as games [ 25 , 16 , 17 ] , robotics [ 8 , 6 ] , autonomous driving [ 9 , 24 ] , and human alignment of large language models [ 21 ] . However, this integration introduces challenges, including increased computational demands and the need for careful hyperparameter tuning. Moreover, RL algorithms often require significant GPU resources, adding to the overall computational complexity. Fig. 1: Performance of linear function approximation Q-learning on a 4\times 4 grid world. Feature sets with high positive correlation result in a bad policy, with longer episode lengths (maximum length capped at 100 ). In contrast, feature sets with lower positive correlation enable faster learning and convergence to shorter episode lengths. Both set of features represent the same current position. Given these challenges, transfer learning has become increasingly important in RL [ 31 ] . The ability to apply knowledge from previously trained models to solve related tasks is more efficient than training new models from scratch. For instance, consider a household robot trained via deep RL to move an object from a source to a designated target. If the target location changes slightly, the existing policy will no longer be effective. Retraining the model from scratch for such a minor variation is inefficient; instead, it would be more practical to leverage the knowledge from the pre-trained model. Additionally, it is not desirable to use complex deep RL algorithms for training the new task, instead one prefers to train a simpler and faster model. A widely adopted method for transfer learning in RL involves training a deep RL neural network on an initial task and then utilizing its hidden layers for a new, related task. In this work, we focus on the Deep Q-Learning (DQN) [ 16 ] , a popular deep RL algorithm. The DQN model takes the state of the environment as input and predicts Q-values for all possible actions, representing the expected cumulative reward for each action. An interesting interpretation of DQN is to view it as the network that transforms the input state into a more meaningful and compact representation before combining this representation linearly to generate the Q-values [ 15 ] . Specifically, the output of the final hidden layer can be viewed as a refined state representation. We propose leveraging these refined state representations from a trained DQN model and using them as state features to train a linear function approximation Q-learning for the new task, thereby facilitating more efficient and simpler transfer learning. However, transferring these improved state representations is not without its challenges. A key requirement for effective linear function approximation is that the state features must be sufficiently distinguishable to ensure faster learning. Unfortunately, deep neural networks are prone to overfitting [ 3 ] , which can lead to highly correlated state representations. We observe that, in standard DQN, the outputs from the final hidden layer, which represent the state features, can exhibit strong positive correlations. This becomes a significant challenge for the linear function approximator, as it becomes difficult to learn optimal weights for the new task. For instance, in Figure 1 , we demonstrate the impact of highly correlated features in a grid world environment, where the objective is to reach the target state from an initial state in fewer steps. The linear function approximator struggles to learn even suboptimal behaviour when the state features are highly correlated (average correlation across the states is 0.89 ), whereas features with less correlation (average correlation of -0.07 ) allow for learning the optimal path more effectively. To address this issue, we introduce a regularization term to the DQN loss function during training that penalizes positive correlations between state features. This modification ensures that the trained DQN not only performs well on the original task but also produces less positively correlated state features. While the goal of function approximation is to generalize effectively across states, this regularization term ensures that positive correlations are preserved only where necessary. Consequently, the resulting state features are more distinguishable and can be leveraged for transfer to new tasks. To this end, we first present an interpretation of DQN as a network that transforms raw states into more refined state representations, which are then linearly combined with weights to produce Q-values. This viewpoint allows us to introduce a regularization term into the DQN loss function to reduce feature correlation. To optimize this new loss function, we propose an algorithm to effectively train with this modified loss function. Using the learned state features, we then apply linear function approximation (LFA) Q-learning. Our results demonstrate that LFA Q-learning performs effectively in complex environments, such as Lunar Lander and MinAtar, which were previously considered unlearnable with linear methods alone. Additionally, we show that the model adapts well to new environments, even with slight modifications in reward structures, thus achieving transfer learning with significantly reduced computational overhead. II Related Works In this section, we review works from the literature, categorizing them into three key related areas: representation learning and feature decorrelation, transfer learning, and linear function approximation. Decorrelation and Representation Learning . Sparse Coding [ 27 , 26 ] is an effective but computationally demanding way of designing feature representations in RL. [ 13 ] has shown that sparse representations can help address environments that are too

Read the complete text