Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following

Yuxiao Yang, Shenao Zhang, Zhihan Liu, Huaxiu Yao, Zhaoran Wang · 2024

Read Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following by Yuxiao Yang, Shenao Zhang, Zhihan Liu, Huaxiu Yao, Zhaoran Wang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-19562.txt.

Introduction

Yuxiao Yang and four co-authors build a robot task planner that learns from its own hindsight, in a 2024 paper. Home robots that follow spoken orders need plans, and language models can draft them. Plans learned by copying experts crack when one off step lands the robot in a state no expert visited. The team frames planning as choice under partial sight and gives the planner few examples, not hordes.

A hindsight module re-reads each attempt, wins and slips alike, to squeeze lessons for the next try. An adaptation part folds those lessons back into the plan as the robot works, closing the loop. On a standard home-task benchmark, the few-shot planner matched and at times beat agents trained on full expert sets. The authors say feedback, not just imitation, is what makes planners sturdy.

Catalog record

CREATORYuxiao Yang, Shenao Zhang, Zhihan Liu, Huaxiu Yao, Zhaoran Wang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-19562

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hindsight Planner: A Closed-Loop Few-Shot Planner for Embodied Instruction Following Yuxiao Yang, Shenao Zhang, Zhihan Liu, Huaxiu Yao, Zhaoran Wang Abstract This work focuses on building a task planner for Embodied Instruction Following (EIF) using Large Language Models (LLMs). Previous works typically train a planner to imitate expert trajectories, treating this as a supervised task. While these methods achieve competitive performance, they often lack sufficient robustness. When a suboptimal action is taken, the planner may encounter an out-of-distribution state, which can lead to task failure. In contrast, we frame the task as a Partially Observable Markov Decision Process (POMDP) and aim to develop a robust planner under a few-shot assumption. Thus, we propose a closed-loop planner with an adaptation module and a novel hindsight method, aiming to use as much information as possible to assist the planner. Our experiments on the ALFRED dataset indicate that our planner achieves competitive performance under a few-shot assumption. For the first time, our few-shot agent’s performance approaches and even surpasses that of the full-shot supervised agent. 1 Introduction With the development of AI and robotics, many previous works have combined them to handle Embodied Instruction Following (EIF). Among them, the Action Learning From Realistic Environments and Directives (ALFRED) benchmark ( Shridhar et al., 2020 ) is particularly challenging because it requires an agent to learn a long-horizon policy that maps egocentric images and language instructions into a sequence of actions. In each task, the agent will be given a natural instruction (e.g. “Put a heated mug down on a table”) and an egocentric visual observation at each step. The agent is required to output low-level actions (e.g. MoveAhead, RotateRight, etc.) based on the observation to complete the task. These tasks are usually challenging due to the sparse reward settings. For such a reason, many works have adopted a hierarchical structure to deal with it ( Song et al., 2023 ; Min et al., 2021 ; Blukis et al., 2021 ; Kim et al., 2024 ) . The high-level module decomposes the whole task into several sub-goals, the low-level module outputs actions to finish each sub-goal. Previously, sub-goal planners are trained on human-annotated dataset through supervised learning. However, they require large amounts of data and often lack robustness ( Min et al., 2021 ; Blukis et al., 2021 ; Kim et al., 2024 ) . With recent advancements in Large Language Models (LLMs), many studies have explored using LLMs as sub-goal planners, utilizing their in-context learning abilities ( Song et al., 2023 ; Shin et al., 2024 ; Ahn et al., 2022 ) . Although these methods have achieved competitive performance under the few-shot assumption, a critical limitation is that these approaches all study the problem from a supervised learning perspective. They merely attempt to imitate the ground truth trajectories, which results in a lack of robustness within their agents. EIF benchmarks, on the other hand, require long-horizon planning ability. For example, the task “Put a warmed apple in the fridge” requires 12-step planning. Assuming that after applying in-context learning, the distribution of the agent’s output actions becomes closer to that of the Oracle, with an accuracy of 0.9, the overall accuracy of the entire planning task decreases to 0.9^{12}=0.28 . Traditionally, a large amount of data is required to mitigate such an issue ( Blukis et al., 2021 ; Kim et al., 2024 ) . However, under the few-shot assumption, in-context learning methods rely heavily on the reasoning ability of pretrained LLMs ( Brown et al., 2020 ; Dong et al., 2024 ) . The hallucination problem of LLMs ( Zhang et al., 2023 ) suggests that supervised methods through in-context learning are limited. To address this issue, we approach the ALFRED task ( Shridhar et al., 2020 ) as a Partially Observable Markov Decision Process (POMDP), where the planner makes decisions based on its current state. Each task begins with a natural language description. At each step, the planner receives an egocentric RGB image and returns a high-level sub-goal. The planner can only receive reward signals (Success or Fail) at the end of the task. There are three major challenges in building a robust planner: (1) The sparse reward settings make it difficult for the planner to learn and make accurate decisions. (2) The planner can only receive an egocentric picture and cannot detect the whole state. (3) Under the few-shot assumption, the planner cannot obtain enough information from trajectories. For the first problem, we adopt an actor-critic framework ( Liu et al., 2024 ) which consists of two actors, one critic, and one generator. At each step, the planner receives a new state and performs a tree search with the actors and generator to plan future trajectories, rather than directly outputting a sub-goal. The critic is then used to select the best rollout and return its initial action. Thus, the planner can optimize the output over the long horizon to address the issue of sparse reward. For the second difficulty, we design an adaptation module instantiated by LLMs. Upon receiving an egocentric image, the adaptation module aims to predict the invisible latent PDDL variables of the task, which could help the planner better understand the environment. For the third challenge, we propose a novel hindsight method. It collects suboptimal trajectories from the agent in the training environment and relabels them to complete the task. This approach provides the planner with additional information. During the deployment phase, we prompt one actor with ground truth samples, while the other actor is prompted with hindsight samples. Thus, the relabeled trajectories can guide the planner in adjusting its policy when incorrect actions are proposed and executed. Figure 1: Left : The illustration of the Hindsight Planner: at each time step t , the planner receives a partial observation y^{t} from the environment. The adaptation module estimates the latent variable and concatenates it with y^{t} to produce the complete state x_{t} . \texttt{Actor}_{\texttt{hind}} and \texttt{Actor}_{\texttt{gt}} are prompted with different samples and make decisions. The Critic is utilized to evaluate the actions. The best rollout (x_{t},a_{t}^{*},x_{t+1}^{*},a_{t+1}^{*}\ldots) is selected, and a_{t}^{*} is returned. Right : An example of the relabeling process for the \texttt{Actor}_{\texttt{hind}} : after collecting a suboptimal rollout, the LLM is prompted to generate a reflection on the previously taken actions. Following this reflection, the LLM is then prompted to complete the suboptimal rollout. In summary, our contributions are threefold: (1) We study ALFRED ( Shridhar et al., 2020 ) from a POMDP perspective for the first time and propose a closed-loop actor-critic planner to solve it. (2) We propose a novel hindsight prompting method and demonstrate that our method is theoretically superior to previous approaches. (3) Experiments on ALFRED ( Shridhar et al., 2020 ) show that our method achieves state-of-the-art performance under few-shot assumptions. Specifically, the success rates for the “Test Seen” and “Test Unseen” splits are 25.51 and 18.77 , respectively, representing a 60\% and 39\% improvement over the previous baseline. 2 Related Work 2.1 Large Language Model (LLM) and In-Context Learning (ICL) Large language models (LLMs) have shown incredible reasoning ability ( Vaswani et al., 2023 ; Wei et al., 2022 ; Touvron et al., 2023 ; OpenAI et al., 2024 ) across

Read the complete text