The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data
Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar · 2024
Read The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data by Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06877.txt.
Introduction
Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, and Mihaela van der Schaar train agents that follow word-based goals. Their 2024 paper tackles a hard setting: labeled data are scarce and live trials are not practical. Standard reinforcement learning often fails on new goals and new states. Their pipeline, called TEDUO, learns from unlabeled data gathered offline.
Large language models serve two roles in the pipeline. They enrich the offline data with richer notes, and they act as agents that follow instructions. Tests show the method learns robust policies with little data. It solved tasks that standard methods and raw language models could not reach on their own. The work targets symbolic settings where goals are stated in words.
Catalog record
| CREATOR | Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-06877
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Synergy of LLMs & RL Unlocks Offline Learning of Generalizable Language-Conditioned Policies with Low-fidelity Data Thomas Pouplin, Katarzyna Kobalczyk, Hao Sun, Mihaela van der Schaar Abstract Developing autonomous agents capable of performing complex, multi-step decision-making tasks specified in natural language remains a significant challenge, particularly in realistic settings where labeled data is scarce and real-time experimentation is impractical. Existing reinforcement learning (RL) approaches often struggle to generalize to unseen goals and states, limiting their applicability. In this paper, we introduce TEDUO , a novel training pipeline for offline language-conditioned policy learning in symbolic environments. Unlike conventional methods, TEDUO operates on readily available, unlabeled datasets and addresses the challenge of generalization to previously unseen goals and states. Our approach harnesses large language models (LLMs) in a dual capacity: first, as automatization tools augmenting offline datasets with richer annotations, and second, as generalizable instruction-following agents. Empirical results demonstrate that TEDUO achieves data-efficient learning of robust language-conditioned policies, accomplishing tasks beyond the reach of conventional RL frameworks or out-of-the-box LLMs alone. † † affiliationnotice: Equal contribution 1 Introduction Motivation. Enabling AI agents to follow human-provided natural language instructions is a critical step toward building intelligent systems that can perform complex, multi-step tasks in dynamic environments. However, existing methods often require vast quantities of annotated data or rely on costly online interactions with the environment to train effective policies. Such requirements limit their scalability to real-world scenarios where collecting annotated or interaction data is infeasible. Moreover, current RL-based agents often struggle to generalize to new goals and environments beyond their training data ( Yang et al., 2023 ) . These challenges underline the need for a change of focus in the area of instruction-following agents: from reliance on online interactions in environments with rich forms of feedback to the learning of generalizable policies from unlabeled observational data. We illustrate the problem setting considered with the following hypothetical scenario: Imagine recording a human using their smartphone in daily tasks, such as text messaging or calendar management. How can we train an an AI assistant following natural language instructions, e.g. “Book a restaurant for 7 PM” or “Send an email to John.” , by just recording the actions taken by the human and not having to manually label each state or action recorded? How can we make the learned policies generalize to commands that have not been explicitly observed in the data? Problem setting. We formalize our setup by framing it as an offline language-conditioned policy learning problem. We assume access to a pre-collected dataset of state-action transitions, {\mathcal{D}} consisting of triplets (x,a,x^{\prime}) , where x and x^{\prime} belong to an observable state space {\mathcal{X}} , and a represents actions within an action space {\mathcal{A}} . To enable the grounding of the environment dynamics with the space of natural language we additionally require: 1) that the individual states and actions are representable in a textual format; 2) access to an unordered list of goals, {\mathcal{G}} expressed in natural language that are plausibly achievable within the environment of interest. We posit that both {\mathcal{D}} and {\mathcal{G}} are often easy to obtain by simply recording agents interact with the environment and curating a list of natural language commands corresponding to the tasks typically performed within that environment. Without constraining assumptions regarding the optimality of the data collection policy with respect to the goals {\mathcal{G}} , nor access to the ground-truth state-transition dynamics or rewards, our aim is to learn a language-conditioned policy, \pi^{*} , that can determine optimal actions with respect to new states x\notin{\mathcal{D}} and new goals g^{*}\not\in{\mathcal{G}}^{tr} , where {\mathcal{G}}^{tr}\subset{\mathcal{G}} is the subset of goals used for training (see section 2 for details). Challenges. The task of learning \pi^{*} from {\mathcal{D}} and {\mathcal{G}}^{tr} alone might seem impossible without resorting to human supervision. We can immediately identify the following challenges: C1) Unlabeled data. The dataset {\mathcal{D}} lacks explicit labels linking states x\in{\mathcal{X}} to the goals g\in{\mathcal{G}} . Nor does it include any rewards indicating the optimality of actions in relation to these goals. C2) Limited exploration. We are in an offline setup with our knowledge of the environment dynamics being constrained to the state-action transitions observed in {\mathcal{D}} . C3) Unknown data collection. We make no assumptions regarding the optimality of the data collection policy with respect to the training or testing goals. The actions in {\mathcal{D}} could be entirely random or generated by policies aimed at solving goals with an unknown relationship to those in {\mathcal{G}} . C4) Generalization. Beyond solving goals from {\mathcal{G}}^{tr} and taking optimal actions in previously observed states x\in{\mathcal{D}} , we want our agent to generalize to new states and language commands corresponding to genuinely novel goal states. Solution: LLMs & RL synergy. Recent advances in LLMs offer a promising solution to these challenges. LLMs, pre-trained on vast amounts of Internet data, possess the requisite prior knowledge to understand natural language and follow simple instructions. However, while LLMs excel at general language comprehension, their ungrounded knowledge is insufficient for executing complex, multi-step decision-making tasks in dynamic environments ( Finn, 2024 ; Szot et al., 2024 ) . In this paper, we propose a novel, sequential training pipeline for offline language-conditioned policy learning— TEDUO: Teaching the Environment Dynamics from Unlabeled Observations . As displayed in Figure 1 , TEDUO distills knowledge of the environment dynamics into a pre-trained LLM through supervised fine-tuning (SFT). This knowledge is obtained by learning optimal policies with traditional RL, based on the offline dataset augmented with LLM-generated labels and optional state abstractions. Within TEDUO, LLMs fulfill the dual role of cheap data enhancers and flexible generalizers , elevating conventional RL to address challenges C1-C4. Contributions. We categorize our contributions as having a significant impact both for the fields of LLMs and RL. Significance for the LLM Community: \blacktriangleright Grounding LLMs for Multi-Step Decision Making: In line with previous research, we demonstrate that standalone LLMs surprisingly fail even in simple multi-step decision-making tasks. We identify this limitation as stemming from the lack of grounding in environment dynamics. Crucially, we show that such grounding can be successfully achieved through SFT, enabling LLMs to integrate their background knowledge with actionable policies. \blacktriangleright Core Skill Acquisition: Our analysis reveals that fine-tuned LLMs acquire core decision-making skills rather than merely memorizing optimal actions. This highlights the potential of LLMs to generalize effectively to previously unsolved tasks. Significance for the RL Community: \blacktriangleright Offline Learning from Low-fidelity Data : We introduce the first method to enable learning generalizable language-conditioned policies using only an unlabeled dataset of state-action transitions and an unpaired set of language commands. This addresses a critical bottleneck of conventional RL by eliminating the need for expensive labeled datasets or real-time experimentation. \blacktriangleright Enhanced Generalization and Data Efficiency: TEDUO significantly improves both the generalization capacity and data efficiency of offline training, outperforming conventional RL approaches. We provide empirical evidence that our method scales effectively and facilitates robust policy learning in symbolic environments. Figure 1: Overview of TEDUO. \raisebox{-0.9pt}{1}⃝ The unlabeled dataset of state-action transitions is pre-processed with LLM-automated hindsight labeling and state abstraction. \raisebox{-0.9pt}{2}⃝ The resulting labeled dataset of abstract state transitions is used as the input to an offline RL algorithm to learn the optimal goal-conditioned policies for the finite