Knowledge Transfer in Model-Based Reinforcement Learning Agents for Efficient Multi-Task Learning Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 565
Dmytro Kuzmenko, Nadiya Shvai · 2025
Read Knowledge Transfer in Model-Based Reinforcement Learning Agents for Efficient Multi-Task Learning Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 565 by Dmytro Kuzmenko, Nadiya Shvai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05329.txt.
Introduction
Dmytro Kuzmenko and Nadiya Shvai present a way to shrink large game-playing AI agents in this 2025 paper. Their agents learn by building an inner model of the world, which works well but takes huge networks. Big models are hard to run on small devices with tight limits.
The method teaches a small agent to copy a large multi-task one. The large agent had hundreds of millions of parts; the small copy had about one million. On a thirty-task test set, the small copy scored far above a same-size agent trained from scratch. A further squeeze that halves the stored size kept the scores intact. The authors say the approach can bring strong agents to robots and other limited hardware.
Catalog record
| CREATOR | Dmytro Kuzmenko, Nadiya Shvai |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 9 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-05329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Knowledge Transfer in Model-Based Reinforcement Learning Agents for Efficient Multi-Task Learning Conference: Proc. of the 24th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2025); May 19 – 23, 2025; Detroit, Michigan, USA 565 Dmytro Kuzmenko, Nadiya Shvai Abstract . We propose an efficient knowledge transfer approach for model-based reinforcement learning, addressing the challenge of deploying large world models in resource-constrained environments. Our method distills a high-capacity multi-task agent (317M parameters) into a compact 1M parameter model, achieving state-of-the-art performance on the MT30 benchmark with a normalized score of 28.45, a substantial improvement over the original 1M parameter model’s score of 18.93. This demonstrates the ability of our distillation technique to consolidate complex multi-task knowledge effectively. Additionally, we apply FP16 post-training quantization, reducing the model size by 50% while maintaining performance. Our work bridges the gap between the power of large models and practical deployment constraints, offering a scalable solution for efficient and accessible multi-task reinforcement learning in robotics and other resource-limited domains. Y. Vorobeychik, S. Das, A. Nowe (eds.) Extended Abstract 1. Introduction Reinforcement learning (RL) has achieved significant progress in diverse domains, yet it remains challenging to efficiently train agents for multiple tasks in resource-constrained environments. Model-based RL approaches, such as TD-MPC2 ( Hansen et al. 2024 ) , leverage large world models for superior performance and generalization but require substantial computational resources, limiting real-world applicability. Our work addresses this by focusing on optimizing large model-based RL agents for efficient multi-task learning through knowledge distillation and model compression. Specifically, we transfer knowledge from high-capacity TD-MPC2 models to compact 1M parameter backbones suitable for deployment in resource-limited scenarios. Our method builds on traditional teacher-student distillation Hinton et al. 2015 ; Rusu et al. 2015 ; Teh et al. 2017 ; Parisotto et al. 2015 ; Czarnecki et al. 2019 and incorporates quantization techniques ( Micikevicius et al. 2018 ) , creating a pipeline for lightweight RL models. Using the MT30 benchmark Hansen et al. 2024 ; Tassa et al. 2020 , our FP16-quantized model achieves state-of-the-art performance (28.45 normalized score), surpassing the original model (+48.5%) and outperforming models trained from scratch (+2.77%). By combining extended distillation periods, optimized batch sizes, and quantization, our approach bridges the gap between the impressive capabilities of large-scale RL models and their deployment in robotics and other real-world applications. 2. Methods Our approach employs a teacher-student distillation framework, adapting it to the specific challenges of model-based RL (Figure 1). The 317M-parameter TD-MPC2 model servers as the teacher, representing the largest available checkpoint from the TD-MPC2 ( Hansen et al. 2024 ) , and the respective 1M-parameter checkpoint serves as the student. We build upon the original TD-MPC2 loss functions (consistency, reward, and value losses) by introducing an additional reward distillation loss. This new loss is computed as the mean squared error (MSE) between the rewards predicted by the teacher and student models: L_{\text{distill}}=\text{MSE}(R_{\text{teacher}}(s,a),R_{\text{student}}(s,a)) where R_{\text{teacher}} and R_{\text{student}} are the reward predictions of the teacher and student models, respectively. An additional reward distillation coefficient ( d_coef ) is introduced to balance the original TD-MPC2 loss with our new distillation loss: total\_loss=original\_loss+d_{\text{coef}}*distillation\_loss The d_coef acts as a hyperparameter controlling the influence of the teacher model’s knowledge on the student model’s learning process. We empirically find that values close to 0.5 yield the best results, with 0.4 being optimal for most training setups (Table 1). The normalized score is used as our primary evaluation metric, consistent with ( Hansen et al. 2024 ) . Each task is scored on a scale of 1 to 1000, with the average sum divided by the number of tasks, resulting in a final range of 1-100. For training, we take a 317M parameter TD-MPC2 model checkpoint pretrained on the MT30 dataset. We then train the 1M parameter student model using our distillation approach. The student is trained on the same dataset but with the additional reward distillation loss. We explore different distillation periods, from 200,000 steps to 1,000,000 steps, and various batch sizes to understand the impact of extended distillation on performance. Lastly, FP16 quantization is applied to reduce model size with minimal impact on task performance, highlighting the method’s scalability for resource-constrained deployments. For the MT30 benchmark, we utilize the full available dataset of 690,000 episodes (345,690,000 transitions). We conduct our experiments on a desktop PC with a single RTX 3060 GPU (12GB VRAM). Figure 1. Our distillation approach consists of two main loss function components: the original loss from TD-MPC2 (in red), calculated as a linear combination of consistency, reward, and value losses ( \alpha denotes a scaling coefficient for each respective loss); and the distillation loss (in teal) that calculates MSE between student’s (green) and teacher’s (blue) rewards produced from inferring the same state-action pair. The total loss is a linear combination of both losses, with the distillation loss scaled by the d_coef . In our scenario, the student is trainable while the teacher’s weights are frozen. The figure depicts a diagram of our proposed distillation pipeline. A pretrained teacher and a trainable student are used to infer rewards for reward distillation loss calculation. The original loss from TD-MPC2 is preserved, and the total loss is a linear combination of both losses, with the distillation loss scaled by the \textit{d\_coef}. 3. Results One impactful finding of our study is the substantial improvement achieved through distillation, particularly with extended training and reduced batch size. Short-term distillation results showed a significant improvement over the baseline. The distilled model with d_coef = 0.4 achieved a normalized score of 17.85, compared to the baseline score of 14.04, representing a 27.1% improvement. Extended distillation yielded similar results. The distilled model ( d_coef = 0.45) achieved a normalized score of 28.12, compared to the model trained from scratch with a score of 27.36, representing a 2.77% improvement. Our findings indicate that for our distillation approach in the context of TD-MPC2, a batch size of 256 offers an optimal balance of performance, efficiency, and hardware compatibility (Table 2). Comparing distillation from different teacher model sizes revealed the value of using a larger, more knowledgeable teacher model. Distillation from the 48M parameter teacher resulted in a score of 13.61, while distillation from the 317M parameter teacher achieved 17.85, representing a 31.2% relative improvement. Our experiments reveal that incorporating next-state latent distillation alongside reward distillation presents a significant challenge due to the dimensional mismatch between teacher (1376-dim) and student (128-dim) models. This mismatch necessitated dimensionality reduction techniques, such as linear projection and PCA, which introduced substantial information loss and led to poor generalization. Linear projection yielded a normalized score of 7.69, while PCA marginally improved this to 8.78, but both fell far short of the 14.04 achieved with reward-only distillation. The additional computational overhead of PCA, combined with its limited gains, further underscored its inefficiency. These findings suggest that next-state prediction, which relies on abstract intermediate representations, often deviates from the distilled model’s optimal structure, making it suboptimal for our setup. In contrast, reward prediction directly aligns with task-specific performance, simplifying the learning process and enabling better outcomes, particularly in reward-oriented tasks like pendulum-swingup and cup-catch . Table 1. Impact of d_coef on student’s performance, 200K distillation steps with 317M parameter teacher and batch size of