TANGO: Training-free Embodied AI Agents for Open-world Tasks
Filippo Ziliotto, Tommaso Campari, Luciano Serafini, Lamberto Ballan · 2024
Read TANGO: Training-free Embodied AI Agents for Open-world Tasks by Filippo Ziliotto, Tommaso Campari, Luciano Serafini, Lamberto Ballan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10402.txt.
Introduction
Filippo Ziliotto and co-authors let a language model conduct robot skills like an orchestra, with no task training at all, in a 2024 paper. Language models can already stitch vision modules into programs for still images. TANGO extends the trick to embodied agents that move through the world. Its base skills are a point-goal navigator and a memory-guided explorer; the model composes them per task from a few examples in the prompt.
Three hard tests followed: finding named objects, lifelong navigation mixing cues, and answering questions about the surroundings. The composed agent hit state-of-the-art results in zero-shot settings. Training-free composition, the authors argue, scales better than one model per task.
Catalog record
| CREATOR | Filippo Ziliotto, Tommaso Campari, Luciano Serafini, Lamberto Ballan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10402
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TANGO: Training-free Embodied AI Agents for Open-world Tasks Filippo Ziliotto, Tommaso Campari, Luciano Serafini, Lamberto Ballan Abstract Large Language Models (LLMs) have demonstrated excellent capabilities in composing various modules together to create programs that can perform complex reasoning tasks on images. In this paper, we propose TANGO, an approach that extends the program composition via LLMs already observed for images, aiming to integrate those capabilities into embodied agents capable of observing and acting in the world. Specifically, by employing a simple PointGoal Navigation model combined with a memory-based exploration policy as a foundational primitive for guiding an agent through the world, we show how a single model can address diverse tasks without additional training. We task an LLM with composing the provided primitives to solve a specific task, using only a few in-context examples in the prompt. We evaluate our approach on three key Embodied AI tasks: Open-Set ObjectGoal Navigation, Multi-Modal Lifelong Navigation, and Open Embodied Question Answering, achieving state-of-the-art results without any specific fine-tuning in challenging zero-shot scenarios. 1 Introduction Figure 1 : We introduce TANGO , a modular neuro-symbolic system for compositional embodied visual navigation. Given a few examples of natural language instructions and the corresponding programs composed of action primitives, TANGO can generate executable programs, enabling the agent to perform multiple tasks within a 3D environment. Large Language Models (LLMs)have gained significant attention in the field of AI due to their remarkable capability to generalize across unseen tasks Brown et al. 2020 ; Team et al. 2023 ; Jiang et al. 2023 ; Touvron et al. 2023 . Systems like VisProg and ViperGPT Gupta and Kembhavi 2023 ; Surís et al. 2023 demonstrated strong performance on a broad range of vision-and-language tasks by just relying on a few numbers of in-context examples to generate complex compositional programs without requiring any specific training. Nevertheless, the evaluation of these concepts to Embodied AI, particularly for navigation tasks, remains largely unexplored. This approach may be crucial in building agents capable to navigate and operate efficiently in unfamiliar environments. Previously, methods such as Neural Module Networks (NMN) Andreas et al. 2016 ; Hu et al. 2017 have demonstrated promising compositional properties for high-level tasks like visual question answering, via end-to-end training of networks and specialized, differentiable neural modules. However, they required the combination of semantic parsers and predefined templates to learn to solve the compositional tasks. Alternative approaches leverage LLMsthrough predefined task modules or APIs for action execution Zhou et al. 2024 ; Wu et al. 2023 ; Huang et al. 2022a ; Huang et al. 2022b ; Liang et al. 2023 . While effective, these methods often lack a compositional structure to address multiple tasks or primarily focus on manipulation and robotic planning, with limited emphasis on navigation. In this work, we present TANGO (see Figure 1 ) a novel neuro-symbolic compositional approach which utilizes primitives and employs a LLMas a planner to sequence these primitives within photorealistic 3D environments, where agents must perceive and act. This framework integrates high-level planning with low-level action execution, without the need for training. TANGO makes use of diverse modules designed for visual navigation and question-answering tasks, resulting in a system that seamlessly addresses both challenges while achieving SoA performance without requiring any prior adjustments. By providing a few in-context examples that show how to tackle multiple tasks, TANGO is capable of generalizing to the specific task at hand. This is facilitated by the LLM, which effectively combines the individual modules available within the TANGO system. Moreover, it extends Yokoyama et al. 2024a exploration policy by incorporating a memory mechanism in the form of a stored feature map that retains information about previously explored areas, supporting for efficient life-long navigation tasks. To demonstrate the flexibility of the proposed framework, TANGO has been tested on three popular benchmarks: namely, i ) Open-Vocabulary ObjectGoal Navigation Yokoyama et al. 2024b , ii ) Multi-Modal Lifelong Navigation Khanna et al. 2024 , and iii ) Embodied Question Answering Majumdar et al. 2024 ; Das et al. 2018 . Results match or surpass previous approaches, without requiring specialized training. Summing up, the contributions of this paper are as follows: • Introduction of a neuro-symbolic compositional LLM-based framework for EAI leveraging specialized primitive modules. • Demonstration of robust generalization capabilities across multiple tasks without the need for any specific training or fine-tuning. • Extension of the exploration policy presented in Yokoyama et al. 2024a to multi-goal scenarios through the incorporation of a memory mechanism stored as a feature vector map. • Achievement of state-of-the-art results, underscoring the effectiveness of the proposed framework. 2 Related Works Our work takes inspiration from modular approaches for visual tasks, like the seminal Neural Module Networks Andreas et al. 2016 ; Hu et al. 2017 , as well as the recent VisProg Gupta and Kembhavi 2023 and ViperGPT Surís et al. 2023 frameworks. The main contribution of this paper is to extend these ideas to embodied visual navigation, that usually require heavy end-to-end learning, although modular approaches have recently shown promising results Gervet et al. 2023 ; Campari et al. 2022 . Therefore, we discuss prior work in the area of embodied AI, the use of language models in robotics, and program generation approaches for image recognition tasks. Embodied AI and Visual Navigation. The field of Embodied AI has recently undergone a paradigm shift, fuelled by the emergence of highly efficient simulators Savva et al. 2017 ; Savva et al. 2019 ; Kolve et al. 2017 ; Shen et al. 2021 ; Puig et al. 2023 . These simulators enable processing numerous parallel simulations in photorealistic indoor environments, facilitating large-scale testing that was otherwise challenging in classical robotics. Alongside these simulators, a large variety of tasks (and benchmarks) has been proposed: PointGoal Navigation Wijmans et al. 2019 , where an agent navigates from point A to point B in an unknown environment; ObjectGoal Navigation Batra et al. 2020 , requiring the agent to locate and navigate to an object in the scene; Instance-Image Goal Navigation Krantz et al. 2022 , akin to ObjectGoal Navigation, but in which the agent should find a specific object depicted in a given image; Multi-Modal Lifelong Navigation Khanna et al. 2024 involving navigating to a sequence of target objects that can be specified through labels, images, or textual descriptions; Embodied Question Answering Majumdar et al. 2024 ; Das et al. 2018 , in which an agent navigates an environment to gather information needed to answer a question, and Vision and Language Navigation (VLN) Anderson et al. 2018 . Various approaches have been proposed to tackle these tasks, primarily falling into three categories: end-to-end Ye et al. 2020 ; Ye et al. 2021 ; Campari et al. 2020 , modular Chaplot et al. 2019 ; Chaplot et al. 2020b ; Chaplot et al. 2020a ; Ramakrishnan et al. 2022 ; Krantz et al. 2023 ; Raychaudhuri et al. 2024 and LLM-based Zhou et al. 2024 ; Dorbala et al. 2023 ; Zhou et al. 2025 ; Yu et al. ; Li et al. 2024 . However, the main limitation of these paradigms is their dependence on task-specific architectures. In the case of end-to-end approaches, a model is trained for a specific