FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning

Alessandro Capurso*, Elia Piccoli*, Davide Bacciu · 2025

Read FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning by Alessandro Capurso*, Elia Piccoli*, Davide Bacciu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20433.txt.

Introduction

Alessandro Capurso, Elia Piccoli, and Davide Bacciu present a 2025 research paper on transfer learning for game-like tasks. Their method, called FAST, helps a trained agent reuse what it learned when it faces a new task. The core problem is picking a good source policy, since a poor pick can slow learning down or even hurt it.

FAST builds a shared code for each task from screen frames and short text notes. It then scores how alike two tasks are and picks the best source policy to copy from. Tests on several racing tracks show results close to learning from scratch, but with far fewer training steps.

Catalog record

CREATORAlessandro Capurso*, Elia Piccoli*, Davide Bacciu
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20433

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FAST: Similarity-based Knowledge Transfer for Efficient Policy Learning Alessandro Capurso*, Elia Piccoli*, Davide Bacciu Abstract Transfer Learning (TL) offers the potential to accelerate learning by transferring knowledge across tasks. However, it faces critical challenges such as negative transfer, domain adaptation and inefficiency in selecting solid source policies. These issues often represent critical problems in evolving domains, i.e. game development, where scenarios transform and agents must adapt. The continuous release of new agents is costly and inefficient. In this work we challenge the key issues in TL to improve knowledge transfer, agents performance across tasks and reduce computational costs. The proposed methodology, called FAST – Framework for Adaptive Similarity-based Transfer, leverages visual frames and textual descriptions to create a latent representation of tasks dynamics, that is exploited to estimate similarity between environments. The similarity scores guides our method in choosing candidate policies from which transfer abilities to simplify learning of novel tasks. Experimental results, over multiple racing tracks, demonstrate that FAST achieves competitive final performance compared to learning-from-scratch methods while requiring significantly less training steps. These findings highlight the potential of embedding-driven task similarity estimations. I Introduction Learning is often thought of as a process rooted in interactions with the environment. Reinforcement Learning (RL) expands on this core concept by viewing learning as a trial-and error process, in which agents engage with the environment, make choices, and receive feedback in the form of reward or penalties. Traditionally, agents are trained from scratch to accomplish a single task, requiring extensive interactions with the environment to achieve proficiency far more than a human would need for comparable tasks. One primary challenge in RL is the substantial computational demands imposed by simulation, where training time and data requirements scale up for complex tasks. In game development and other evolving environments it is expensive and sub-optimal to start at each iteration from zero. For example, during production at each iteration new features are added to the game or the scenario is modified. Given this setting, the following questions come as a logical consequence: what about agents deployed in the previous version? Is it possible to leverage their knowledge and simplify the training of new specialized agents? In attempt to answer these questions, this work challenges the problem of Transfer learning (TL) [ 1 ] in RL. TL focuses on how to carry knowledge from one task, called source , to another problem, called target , where the main objectives are to reduce sample complexity, improve generalization, and speed up adaptation to new environments. However, this approach introduces some challenges as negative transfer and domain adaptation. Negative transfer occurs when knowledge from one domain negatively impacts performance into another. While TL seeks to enforce positive knowledge transfer, it can inadvertently introduce conflicting strategies or biases, impairing learning in specific tasks. This challenge is even more marked in scenarios where source and target task domains differ thus straightforward behavior adaptation is sub-optimal or worst case unsuccessful. Fig. 1 : Schema of the designed similarity metric. Given a set of tasks, environment frames are processed using an autoencoder while goal descriptions are embedded leveraging BERT [ 24 ] . The resulting representations are concatenated for each task and compared using cosine similarity, producing a vector of similarity scores. In order to overcome the problem of negative transfer, a solution is to exploit information about the similarity between different tasks. Given a set of source tasks, agents should be able to choose the one that is more similar to the target task, enforcing positive knowledge transfer and improving the training process. While the idea may seem simple, modeling and quantifying the similarity between different tasks and policies is still an open question [ 1 ] . Recalling the previous example about game development, this would allow to re-use between different iterations of previously trained agents and achieving a faster and adaptable deployment of new bots to newer versions. This work aims to address this challenge by proposing a combination of visual and textual information about tasks to compute a score that models the similarity between scenarios (Fig. 1 ). Based on this information, learning agents can choose the policy of the most similar task from a repository of pre-trained policies. We test our solution on three different target tasks from the driving and racing scenarios in Highway-Env [ 26 ] , achieving competitive performance against a tuned baseline, that learns the task from scratch, while using significantly less training steps. In the evaluation we do not consider only the reward but also other significant metrics that characterize the racing behavior of agents such as distance covered, number of laps completed and average speed. II Background RL problems can be defined as Markov Decision Processes (MDPs), that are tuples \langle\mathcal{S},\mathcal{A},r,P,\gamma\rangle , where \mathcal{S} is the set of states, \mathcal{A} is the set of possible actions, r(s,a):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R} is the reward function, P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1] is the transition function, \gamma\in[0,1) is the discount factor. It is possible to define the return of an agent in an MDP as the sum of discounted rewards from a specific timestep t : G_{t}=R_{t+1}+\gamma R_{t+2}+\dots=\sum_{k=0}^{\inf}\gamma^{k}R_{t+k+1} if \gamma\thickapprox 0 leads to a ”myopic” evaluation, instead, if \gamma\thickapprox 1 leads to a ”far-sighted” evaluation. An RL agent aims to enhance the total reward it gets from the environment over time. The agent’s objective is to learn a policy \pi , that maps states to actions to optimize the cumulative reward in any state. Formally, a policy \pi is a distribution over actions a given states s: \pi(a|s)=P(A_{t}=a|S_{t}=s) The policy depends only on the current state and it is stationary A_{t}\sim\pi(\cdot|s),\forall t>0 . Deep Reinforcement Learning (DRL) combines the principles of RL with deep learning techniques. DRL methods can be categorized into three main approaches: value-based, policy-based, and actor-critic. In value-based methods, the goal is to estimate the optimal action-value function (Q-function) that predicts the future expected reward for taking a specific action in a given state, i.e. DQN [ 2 ] . Policy-based methods, on the other hand, directly optimize the policy without estimating the value function, i.e. PPO [ 3 ] . Actor-critic methods combine elements of both value-based and policy-based approaches, maintaining two sets of parameters: the Critic updates action-value function parameters w and the Actor updates the policy parameters \theta . The critic estimates the value function, which acts as a reference for the actor’s gradient update. By offering more precise estimates of the expected return and decreasing the variance of the updates, it aids in stabilizing the learning process. A notable algorithm in this family, upon which is build our work, is Soft Actor-Critic (SAC) [ 4 ] . This method enhances the standard actor-critic by incorporating an entropy term into the objective function. The inclusion of entropy encourages exploration and adds stability during training. The maximum entropy objective can be expressed as: J(\pi)=\sum_{t=0}^{T}\mathbb{E}_{(s_{t},a_{t})\sim\rho_{\pi}}[r_{t}+\alpha H(\pi(\cdot|s_{t}))] the H(\pi(\cdot|s_{t})) represents the entropy of the policy \pi at state s_{t} , and \alpha is a temperature parameter that balances the reward and entropy terms. A higher \alpha places more emphasis on exploration, while a lower \alpha focuses more on maximizing the reward. III Related works III-A Transfer Learning Over the years, various approaches have been proposed, implemented, and tested

Read the complete text