Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One

Yiwen Song, Qianyue Hao, Qingmin Liao, Jian Yuan, Yong Li · 2025

Read Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One by Yiwen Song, Qianyue Hao, Qingmin Liao, Jian Yuan, Yong Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15306.txt.

Introduction

Yiwen Song and four co-authors combine weak agents into a strong one in twenty twenty-five. Training a good reinforcement learning agent is touchy work. The choice of algorithm, settings, and even random seed can swing results. Ensembles help by merging several agents, but fixed rules like majority voting do not understand the task at hand.

Their method, LLM-Ens, uses a large language model to guide the merger. The model brings task-specific semantic understanding to how agents are combined. That makes the ensemble adapt to the task instead of following a frozen rule. The authors frame the result as many weak agents beating a single strong one. Readers get a smarter way to pool agents that were each trained cheaply.

Catalog record

CREATORYiwen Song, Qianyue Hao, Qingmin Liao, Jian Yuan, Yong Li
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15306

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One Yiwen Song, Qianyue Hao, Qingmin Liao, Jian Yuan, Yong Li Abstract Model ensemble is a useful approach in reinforcement learning (RL) for training effective agents. Despite wide success of RL, training effective agents remains difficult due to the multitude of factors requiring careful tuning, such as algorithm selection, hyperparameter settings, and even random seed choices, all of which can significantly influence an agent’s performance. Model ensemble helps overcome this challenge by combining multiple weak agents into a single, more powerful one, enhancing overall performance. However, existing ensemble methods, such as majority voting and Boltzmann addition, are designed as fixed strategies and lack a semantic understanding of specific tasks, limiting their adaptability and effectiveness. To address this, we propose LLM-Ens , a novel approach that enhances RL model ensemble with task-specific semantic understandings driven by large language models (LLMs). Given a task, we first design an LLM to categorize states in this task into distinct “situations”, incorporating high-level descriptions of the task conditions. Then, we statistically analyze the strengths and weaknesses of each individual agent to be used in the ensemble in each situation. During the inference time, LLM-Ens dynamically identifies the changing task situation and switches to the agent that performs best in the current situation, ensuring dynamic model selection in the evolving task condition. Our approach is designed to be compatible with agents trained with different random seeds, hyperparameter settings, and various RL algorithms. Extensive experiments on the Atari benchmark show that LLM-Ens significantly improves the RL model ensemble, surpassing well-known baselines by up to 20.9%. For reproducibility, our code is open-source at https://anonymous.4open.science/r/LLM4RLensemble-F7EE . 1 Introduction Reinforcement learning (RL) has made unprecedented progress and been recognized as an effective tool for training intelligent agents in solving sequential decision-making tasks sutton2018reinforcement ; franccois2018introduction . Particularly, deep RL has achieved remarkable success in highly complicated tasks, including real-time games silver2017mastering ; vinyals2019grandmaster ; berner2019dota ; ye2021mastering , chip design mirhoseini2021graph , smart city governance hao2021hierarchical ; hao2022reinforcement ; hao2023gat ; zheng2023spatial ; zheng2024survey ; wang2024dyps ; wang2025coopride , where the RL agents are increasingly exhibiting performance surpassing human professionals nowadays. In a wide range of RL research, model ensemble is a useful approach for training effective RL agents. Because a large number of factors will influence the agent’s performance in RL training, it is difficult to consistently obtain reliable RL agents, especially when scaling to more complex tasks. While the choice of RL algorithm is crucial, hyperparameter settings within a single algorithm adkins2024method , such as learning rate and discount factor, can significantly impact the outcome and often require careful tuning and extensive trial-and-error. Moreover, even merely diverse random seeds can lead to completely different outcomes colas2018many . Similar to ensemble methods in machine learning ganaie2022ensemble ; dong2020survey , RL can benefit from combining multiple weak agents, which may individually require minimal hyperparameter tuning and exhibit suboptimal performance in isolation, to produce a powerful agent outperforming any single weak one. This strategy leverages the diversity of the agents to achieve better generalization and more stable performance across different tasks song2023ensemble . However, existing ensemble methods, including majority voting fausser2015neural , weighted aggregation perepu2020reinforcement , and Boltzmann addition wiering2008ensemble , typically rely on fixed, rule-based strategies to combine the agents’ actions. These approaches do not account for the specific characteristics of the task at hand, lacking a semantic understanding of the task’s context. As a result, such methods fail to fully exploit the strengths of each agent to adapt to the complex environments. There exist several challenges in addressing such limitation. First, different tasks have unique characteristics that require tailored ensemble strategies to fully exploit the strengths of each agent. However, RL tasks span a wide range of environments, and designing rule-based strategies for each possible scenario is neither practical nor efficient. Second, RL tasks often involve multiple steps where the state continuously evolving over time. This dynamic nature makes it difficult to rely on pre-defined ensemble strategies, as they may not adapt well to the changing status during the task. Facing these challenges, we propose LLM-Ens , a framework that leverages the contextual understanding and reasoning abilities of large language models (LLMs) to enhance RL model ensemble. The process unfolds in three stages. First, for a given task, we design an LLM to categorize the possible states into distinct “situations” based on the task information, which provide generalized descriptions of the task-specific conditions with semantic understanding, enhancing the adaptability across different tasks. Next, we statistically analyze the performance of each agent that will be used in the ensemble in each categorized situation, identifying their strengths and weaknesses. Finally, during inference, LLM-Ens dynamically adapts to the evolving task by identifying the current situation and selecting the agent that performs best under the identified situation, ensuring the ensemble can respond to changes in the environment flexibly. We conduct extensive experiments on the Atari benchmark bellemare2013arcade ; kaiser2019model to evaluate the performance of LLM-Ens. The results demonstrate that LLM-Ens, through model ensemble, effectively combines multiple weak agents to create a supreme one with optimal performance, outperforming existing RL model ensemble baselines. Our approach is designed to work seamlessly with agents trained using different random seeds, hyperparameter settings, and RL algorithms, maximizing its potential applications. In summary, the main contributions of this work include: • We propose LLM-Ens, a framework that leverages LLMs to generate adaptive and flexible RL model ensemble strategies in different tasks, which effectively combines multiple weak agents into a supreme one with optimal performance. • We conduct extensive experiments to verify the ability of our method to improve RL model ensemble across various RL tasks. The results indicate that our method surpasses established baselines by up to 20.9%. • Our method is compatible with agents trained using different random seeds, hyperparameter configurations, and RL algorithms, ensuring broad applicability across diverse settings. 2 Problem Formulation 2.1 Markov Decision Process (MDP) Markov Decision Process (MDP) is the fundamental framework for reinforcement learning (RL), providing a mathematical structure for modeling sequential decision-making problems where an agent interacts with a dynamic environment. The goal of the agent is to learn an optimal policy that maximizes cumulative rewards over time by making a series of decisions based on observed states. Formally, an MDP is defined as a quintuple (\mathcal{S},\rho,\mathcal{A},P,R) , where \mathcal{S} denotes the set of all possible states that describe the environment, and \rho\in\Delta(\mathcal{S}) represents the probability distribution over the initial states. Here, \Delta(\mathcal{S}) refers to the set of all probability distributions defined over \mathcal{S} . The action space, denoted as \mathcal{A} , includes all possible actions that the agent can take at any given state. The transition dynamics of the environment are modeled by a probability function P:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S}) , which defines the likelihood of transitioning to a new state s^{\prime}\in\mathcal{S} given the current state s\in\mathcal{S} and action a\in\mathcal{A} . Additionally, the reward function R:\mathcal{S}\times\mathcal{A}\to\mathbb{R} specifies the immediate reward obtained when the agent executes action a in state s . At each discrete time step t , the agent observes the current state s_{t}\in\mathcal{S} , selects an action a_{t}\in\mathcal{A} according to its policy, receives a scalar reward

Read the complete text