Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks
Luise Ge, Michael Lanier, Anindya Sarkar, Bengisu Guresti, Chongjie Zhang, Yevgeniy Vorobeychik · 2025
Read Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks by Luise Ge, Michael Lanier, Anindya Sarkar, Bengisu Guresti, Chongjie Zhang, Yevgeniy Vorobeychik (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01885.txt.
Introduction
Ge and co-authors staff a committee of policies instead of betting on one, so a robot can face tasks it never trained on. Multi-task and meta learning blur when tasks differ wildly. Their 2025 approach learns a small committee that, with high probability, holds a near-best policy for whatever task shows up.
The dream version is provably impossible to approximate in general, so the team gives two workable routes. One carries guarantees for low-dimensional task families, and the other is a practical gradient method for the rest. A few-shot learning bound comes along too. On standard robot simulation suites the committee beat multi-task, meta, and clustering rivals, often widely. Code is public.
Catalog record
| CREATOR | Luise Ge, Michael Lanier, Anindya Sarkar, Bengisu Guresti, Chongjie Zhang, Yevgeniy Vorobeychik |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 77 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-01885
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning Policy Committees for Effective Personalization in MDPs with Diverse Tasks Luise Ge, Michael Lanier, Anindya Sarkar, Bengisu Guresti, Chongjie Zhang, Yevgeniy Vorobeychik Abstract Many dynamic decision problems, such as robotic control, involve a series of tasks, many of which are unknown at training time. Typical approaches for these problems, such as multi-task and meta-reinforcement learning, do not generalize well when the tasks are diverse. On the other hand, approaches that aim to tackle task diversity, such as using task embedding as policy context and task clustering, typically lack performance guarantees and require a large number of training tasks. To address these challenges, we propose a novel approach for learning a policy committee that includes at least one near-optimal policy with high probability for tasks encountered during execution. While we show that this problem is in general inapproximable, we present two practical algorithmic solutions. The first yields provable approximation and task sample complexity guarantees when tasks are low-dimensional (the best we can do due to inapproximability), whereas the second is a general and practical gradient-based approach. In addition, we provide a provable sample complexity bound for few-shot learning. Our experiments on MuJoCo and Meta-World show that the proposed approach outperforms state-of-the-art multi-task, meta-, and task clustering baselines in training, generalization, and few-shot learning, often by a large margin. Our code is available at https://github.com/CERL-WUSTL/PACMAN/ . 1 Introduction Reinforcement learning (RL) has achieved remarkable success in a variety of domains, from robotic control ( Lillicrap 2015 ) to game playing ( Xu et al. 2018 ) . However, many real-world applications involve highly diverse sets of tasks, making it impractical to rely on a single, fixed policy. In these settings, both the reward structures and the transition dynamics can vary significantly across tasks. Existing approaches, such as multi-task RL (MTRL) and meta-reinforcement learning (meta-RL), struggle to generalize effectively for diverse and previously unseen tasks. Multi-task RL methods typically train a single policy or a shared representation across tasks ( Vithayathil Varghese & Mahmoud 2020 ) . However, they often face negative transfer, where optimizing for one task degrades performance on others ( Zhang et al. 2022 ) . On the other hand, meta-RL approaches, such as Model-Agnostic Meta-Learning (MAML) ( Finn et al. 2017b ) and PEARL ( Rakelly et al. 2019 ) , aim to enable fast adaptation to new tasks but rely heavily on fine-tuning at test time, which can be computationally expensive and ineffective in environments with high variability in both rewards and transitions, like Meta-World benchmark tasks ( Yu et al. 2020b ) . Several lines of work attempt to address the problem of diverse tasks and negative transfer. The first is to learn policies through MTRL or meta-RL that explicitly take task representation as input ( Lan et al. 2024 ; Grigsby et al. 2024 ; Sodhani et al. 2021 ; Zintgraf et al. 2020 ) . However, such approaches rely on shared parameters, which limit the model’s flexibility. When tasks are highly diverse, using a single neural network— even with a multi-head architecture— may not generalize effectively. Moreover, the greater the task variation, the more data is typically required to train policies that can effectively condition on task embeddings, and the model will often generalize poorly when the number of training tasks is small. Several alternative methods have thus emerged that propose clustering tasks and training distinct policies for each cluster ( Ackermann et al. 2021 ; Ivanov & Ben-Porat 2024 ) . The associated algorithmic approaches commonly leverage EM-style methods that interleave RL and task clustering. In practice, however, they also require a large number of training tasks to be effective and consequently perform poorly when the number of training tasks is small. In addition, if the number of clusters is too small, some clusters may still contain too diverse a set of tasks, with concomitant negative transfer remaining a significant issue. We propose pacman , a novel framework and algorithmic approach for learning policy committees that enables (a) sample efficient generalization (Theorem 3.7 ), (b) few-shot adaptation guarantees that are independent of the dimension of the state or action space (Theorem 3.8 ), and (c) significant improvements in performance compared to 11 state-of-the-art baselines on both MTRL and few-shot adaptation metrics (Section 4 ). The key idea behind pacman is to leverage a parametric task representation which allows clustering tasks in the parameter space first, followed by reinforcement learning for each cluster. The benefit of our approach is illustrated in Figure 1 : even though the single-policy baseline uses a mixture-of-experts approach ( Hendawy et al. 2024 ) , its performance is poor since tasks require fundamentally distinct skills. In contrast, learning a policy committee allows learning custom policies for distinct classes of tasks. Figure 1: Performance on a single task across committee members compared to a MTRL policy. A crucial insight in our approach is to define the objective of clustering as obtaining high coverage of the (unknown) task distribution, as opposed to insisting on a full coverage —that is, achieving near-optimal performance on a randomly drawn task with high probability. This enables us to devise efficient algorithms with provable performance guarantees and polynomial task sample complexity, as well as obtain few-shot adaptation guarantees that depends only on the number of clusters, but not on the size of state and action space. Moreover, while many MTRL and meta-RL problems of interest are non-parametric, we show that we can leverage high-level natural language task descriptions to obtain highly effective embeddings of tasks, akin to Sodhani et al. 2021 , and Hendawy et al. 2024 , but without requiring subtask decomposition. Consequently, our approach can be effectively applied to a broad class of non-parametric MTRL and meta-RL domains as well. In summary, we make the following contributions: • Theoretically Grounded Framework for Learning Policy Committees: We present the first approach for learning policy committees in MTRL and meta-RL domains that offers provable performance guarantees, in addition to practical algorithms. In particular, we provide sample efficient generalization (Theorem 3.7 ) and few-shot adaptation guarantees that are independent of the size of the state or action space (Theorem 3.8 ). • Simple and Effective Policy Committees : pacman is design to handle high task diversity while using a small number of policies to achieve high efficacy. Additionally, our approach leverages LLM-based task embeddings for non-parametric tasks, which provides a highly general and scalable solution for a broad array of environments. • Empirical Validation: We demonstrate the efficacy of pacman through extensive experiments on challenging multi-task benchmarks, including MuJoCo and Meta-World. Our policy committee framework consistently outperforms state-of-the-art multi-task RL, meta-RL, and task clustering baselines in both zero-shot (MTRL) and few-shot (meta-RL) settings, achieving better generalization and faster adaptation across diverse tasks. Related Work: Our work is closely related to three key areas within the broader reinforcement learning literature: multi-task RL, meta-RL, and personalized RL. Multi-Task RL (MTRL): The broad goal of MTRL approaches is to leverage inter-task relationships to learn policies that are effective across multiple tasks ( Yang et al. 2020 ; Sodhani et al. 2021 ; Sun et al. 2022 ) . An important challenge in MTRL is task interference. One class of approaches