DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning
Runpeng Xie ∗1 , Quanwei Wang ∗2 , Hao Hu 3 , Zherui Zhou 4 , Ni Mu 2 , Xiyun Li 5 , Yiqin Yang †1 , Shuang Xu 1 , Qianchuan Zhao 2 , Bo Xu †1 1 The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 Depar · 2025
Read DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning by Runpeng Xie ∗1 , Quanwei Wang ∗2 , Hao Hu 3 , Zherui Zhou 4 , Ni Mu 2 , Xiyun Li 5 , Yiqin Yang †1 , Shuang Xu 1 , Qianchuan Zhao 2 , Bo Xu †1 1 The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 Depar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-19562.txt.
Introduction
Runpeng Xie and nine co-authors tackle a basic snag for agents that follow language commands: the same command can mean two tasks, and two commands can mean the same one. Their method, DAIL, pairs a value-spread policy with a matching module. The module learns which action paths truly match which commands.
The team proves that scoring a spread of values, not a single value, makes tasks easier for the agent to tell apart. On tests with structured and visual inputs, DAIL cleared up command mix-ups better than baseline methods and scored higher. The goal is agents that follow human language well despite its loose ends.
Catalog record
| CREATOR | Runpeng Xie ∗1 , Quanwei Wang ∗2 , Hao Hu 3 , Zherui Zhou 4 , Ni Mu 2 , Xiyun Li 5 , Yiqin Yang †1 , Shuang Xu 1 , Qianchuan Zhao 2 , Bo Xu †1 1 The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 Depar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 108 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-19562
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DAIL: Beyond Task Ambiguity for Language-Conditioned Reinforcement Learning Runpeng Xie ∗1 , Quanwei Wang ∗2 , Hao Hu 3 , Zherui Zhou 4 , Ni Mu 2 , Xiyun Li 5 , Yiqin Yang †1 , Shuang Xu 1 , Qianchuan Zhao 2 , Bo Xu †1 1 The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2 Department of Automation, Tsinghua University 3 Moonshot AI 4 Department of Computer Science and Engineering, Washington University 5 Tecent AI Lab [email protected], [email protected] Abstract Comprehending natural language and following human instructions are critical capabilities for intelligent agents. However, the flexibility of linguistic instructions induces substantial ambiguity across language-conditioned tasks, severely degrading algorithmic performance. To address these limitations, we present a novel method named DAIL (Distributional Aligned Learning), featuring two key components: distributional policy and semantic alignment. Specifically, we provide theoretical results that the value distribution estimation mechanism enhances task differentiability. Meanwhile, the semantic alignment module captures the correspondence between trajectories and linguistic instructions. Extensive experimental results on both structured and visual observation benchmarks demonstrate that DAIL effectively resolves instruction ambiguities, achieving superior performance to baseline methods. Our implementation is available at https://github.com/RunpengXie/Distributional-Aligned-Learning . † † footnotetext: ∗ Equal contribution. † † footnotetext: † Correspondence to Yiqin Yang and Bo Xu. 1 Introduction Artificial agents are anticipated to master diverse skills while effectively interpreting human instructions and generalizing across various tasks. Therefore, comprehension and following of natural language emerges as critical capabilities for agents in this context. For example, language-conditioned agents have achieved remarkable success in robotic manipulation luketina2019survey ; brohan2023rt , text-based environments li2022pre ; carta2023grounding , visual navigation zhou2023language ; hermann2017grounded , and autonomous driving cui2024drive ; roh2020conditional . The fundamental requirement has propelled language-conditioned reinforcement learning (RL) to the forefront of research, which focuses on enabling agents to interpret and execute natural language instructions through RL frameworks. Recent advancements in the language-conditioned RL domain have focused on bridging the gap between linguistic understanding and decision-making processes, aiming to create agents capable of executing complex instructions with human-like adaptability. For example, some works co2018guiding ; bing2023meta integrate language-conditioned policy with trial-and-error learning, significantly improving the performance and sample efficiency in robot task acquisition. Meanwhile, some studies goyal2021pixl2r ; goyal2019using leverage expert demonstrations to map language instructions to reward signals directly to address the issue of sparse rewards in language-conditioned RL. However, linguistic instructions exhibit high flexibility, which induces exponential growth in task space. In this case, identical tasks may have divergent expressions, while distinct tasks share overlapping language instructions. This variability makes language-conditioned RL methods face the significant challenge of task ambiguity, which hinders the agent from discerning the connection between rewards and task objectives, thereby significantly impairing learning efficiency. To solve this issue, we propose a novel method, DAIL, which consists of two main components: a distributional language-guided policy and a trajectory-wise semantic alignment module. Specifically, the distributional policy module bellemare2017distributional estimates the value distribution, preserving more information to aid task discrimination. Theoretically, we analyze the sample complexity required to guarantee task disambiguation and establish that distributional estimation methods are sample-efficient in offline settings. On the other hand, the semantic alignment module constrains the language instruction representations by maximizing the mutual information between trajectories and the instructions, thereby achieving better differentiation across instructions. With theoretical guarantees, the two modules enable precise disambiguation and execution of linguistic instructions, thereby improving learning efficiency. We conduct extensive experiments on both structured observation chevalier2018babyai and visual observation shridhar2020alfred benchmarks. This design is to validate the external validity of the DAIL agent, progressing from less complex structured inputs to more complex and expressive visual observations. The experimental results show that DAIL outperforms the state-of-the-art language-conditioned RL methods in both benchmarks. Further, the visualization analysis demonstrates that DAIL can learn a non-ambiguous task representation compared with baselines. Our main contributions are summarized as follows: • First, we highlight the critical issue of task ambiguity and empirically analyze the limitations of current mainstream methods. We define the task distinction in our setting and analyze the sample complexity to avoid task ambiguity theoretically. • Second, we propose DAIL, a simple yet efficient language-conditioned learning framework, which addresses the task ambiguity issue based on distributional policy and semantic alignment. • Lastly, we conduct extensive experiments to show that DAIL significantly outperforms conventional language-conditioned methods. The results indicate that by improving task discrimination, we can effectively mitigate the task ambiguity issue, thereby broadening the application of language-conditioned RL. 2 Preliminaries Figure 1 : The framework of our method. The key ideas are: (1) use the distributional language-guided policy to aid task discrimination, (2) use the trajectory-wise semantic alignment module to extract task representation. Language-conditioned RL Based on contextual markov decision process (CMDP) hallak2015contextual , we consider Language-conditioned Markov Decision Process (LCMDP) as a model consisting of a tuple \mathcal{M}=(\mathcal{S,A},P,r,\gamma,\mathcal{L},p_{0},p_{l}) , where \mathcal{S} denotes the state space, \mathcal{A} represents the action space, \mathcal{L} is the language instruction space, P(s^{\prime}|s,a,l) represents the probabilistic transition model, r:\mathcal{S\times A\times L}\rightarrow\mathbb{R} is the reward function conditioned on language instructions and \gamma is the discount factor. p_{0} represents the probability distribution of the initial state, and p_{l} denotes the probability distribution of language instruction. We establish language instructions l as task descriptors, and each instruction uniquely specifies a task. Language-conditioned RL aims to obtain a policy \pi(\cdot|s,l) that maximizes the cumulative discounted returns under a specific distribution of language instructions: \displaystyle\pi^{*}={\arg\max}_{\pi}\mathbb{E}_{\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t},l)\right], (1) where s_{0}\sim p_{0}(\cdot),l\sim p_{l}(\cdot),a_{t}\sim\pi(\cdot|s_{t},l) and s_{t+1}\sim P(\cdot|s_{t},a_{t},l) . The temporal difference loss in language-conditioned RL is adapted as follows: \displaystyle L_{\text{TD}}(\theta)= \displaystyle\mathbb{E}_{(s,a,s^{\prime},l)\sim\mathcal{D}}[(r(s,a,l)+\gamma\mathrm{max}_{a^{\prime}}Q_{\hat{\theta}}(s^{\prime},a^{\prime},l)-Q_{\theta}(s,a,l))^{2}] (2) where Q_{\theta}(s,a,l) is the parameterized Q -function conditioned language instruction l , and Q_{\hat{\theta}}(s,a,l) is the target Q -network. Offline RL Due to the high cost of real-time interaction with the environment, we consider the offline learning setting, in which we learns a policy \pi without interacting with an environment. Rather, the learning is based on a dataset \mathcal{D} generated by a behavior policy \pi_{\beta} . One of the major challenges in offline RL is the issue of distributional shift ( fujimoto2019off ) , where the learned policy is different from the behavioral policy. Existing offline RL methods apply various forms of regularization to limit the deviation of the current learned policy: \pi^{*}={\arg\max}_{\pi}\left[J_{\mathcal{D}}(\pi)-\alpha D(\pi,\pi_{\beta})\right], (3) where J_{\mathcal{D}}(\pi) is the cumulative discounted return of policy \pi on the empirical MDP induced by the dataset \mathcal{D} , and D(\pi,\pi_{\beta}) is a divergence measure between \pi and \pi_{\beta} . As for the language-conditioned task, we will provide the language instruction in the evaluation. To make our writing more concise, let \tau be a full trajectory, and \tau_{t} be the trajectory ending at time-step t . Let p_{\mathcal{D}}(\tau,l) represents the joint distribution of trajectory-instruction pairs in the dataset \mathcal{D} . 3 Ambiguity on Language-Conditioned Tasks In practical tasks, language instructions have high flexibility. For example, similar tasks may employ divergent expressions, while distinct tasks share overlapping language instructions. The variability induces exponential growth of the task space. Therefore, when the number of language instructions increases, the agent is required to accurately identify the tasks; otherwise, it will significantly affect the agent’s performance. We name this issue the ambiguity in language-conditioned tasks. We give a