Can LLMs Translate Human Instructions into a Reinforcement Learning Agent’s Internal Emergent Symbolic Representation?

Ziqi MA, Sao Mai NGUYEN, Philippe XU · 2025

Read Can LLMs Translate Human Instructions into a Reinforcement Learning Agent’s Internal Emergent Symbolic Representation? by Ziqi MA, Sao Mai NGUYEN, Philippe XU (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-24259.txt.

Introduction

Ziqi Ma and two co-authors test whether large language models can translate human instructions into the internal symbols of a learning robot agent. Developmental agents that learn in a hierarchy form emergent symbolic representations, which help them plan and generalize. The question is whether a language model can map plain instructions onto those agent specific symbols.

The team evaluates well known models including GPT, Claude, DeepSeek, and Grok with a structured framework. They use symbolic partitions from a hierarchical reinforcement learning algorithm in two ant robot environments, Ant Maze and Ant Fall. Results show the models can partly perform the translation, though performance varies with the partition and environment.

Catalog record

CREATORZiqi MA, Sao Mai NGUYEN, Philippe XU
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE36 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-24259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can LLMs Translate Human Instructions into a Reinforcement Learning Agent’s Internal Emergent Symbolic Representation? Ziqi MA, Sao Mai NGUYEN, Philippe XU Abstract Emergent symbolic representations are critical for enabling developmental learning agents to plan and generalize across tasks. In this work, we investigate whether large language models (LLMs) can translate human natural language instructions into the internal symbolic representations that emerge during hierarchical reinforcement learning. We apply a structured evaluation framework to measure the translation performance of commonly seen LLMs – GPT, Claude, Deepseek and Grok – across different internal symbolic partitions generated by a hierarchical reinforcement learning algorithm in the Ant Maze and Ant Fall environments. Our findings reveal that although LLMs demonstrate some ability to translate natural language into a symbolic representation of the environment dynamics, their performance is highly sensitive to partition granularity and task complexity. The results expose limitations in current LLMs capacity for representation alignment, highlighting the need for further research on robust alignment between language and internal agent representations. [orcid=0009-0006-6775-4263, [email protected], url=https://ziqilovesunshine.github.io, ] [orcid=0000-0003-0929-0019, [email protected], url=http://nguyensmai.free.fr, ] [orcid=0000-0001-7397-4808, [email protected], url=https://perso.ensta-paris.fr/ philippe.xu, ] 1 keywords Emergent Agent ,Symbolic Representation,STAR algorithm ,LLMs ,alignment capacity , † † address: U2IS, ENSTA, IP-Paris, 828 Bd des Maréchaux, 91120 Palaiseau, France † † copyrightyear: 2022 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: 1st Workshop on Interactive Task Learning in Human-Robot co-construction (ITL4HRI) — IEEE RO-MAN 2025: International Conference on Robot and Human Interactive Communication 2025, August 25–29, 2025, Eindhoven, Netherlands † † titlenote: The work is partially supported by Hi! Paris. 1 Introduction Emergent symbolic representation refers to the spontaneous development of internal symbols or abstract concepts within a learning agent, without explicit pre-programming of those symbols but during learning processes that involve interaction with the environment, such as reinforcement learning (RL) or human-in-the-loop learning paradigms Retzlaff et al. 2024 . While most works have focused on symbol emergence for primitive actions, few works have investigated for long-horizon actions, whether the emergent symbols Zadem et al. 2023 ; Ahmetoglu et al. 2022 are aligned with human representations. “Symbol emergence is a critical concept for understanding and creating cognitive developmental systems that are capable of behaving adaptively in the real world and that can communicate with people”, as highlighted in Taniguchi et al. 2018 . The authors in Akakzia et al. 2020 have demonstrated that language instructions can be integrated with emergent symbolic representations of an RL agent in a end-to-end manner. In this work, we investigate the importance of emergent symbols in developmental learning agents to communicate with people. How do these two sets of symbols that have a prior different representations align? What can be used to bridge the gap between internal symbols of developmental learners and natural language symbols used by human instructors? Large Language Models (LLMs) have emerged as a powerful tool in natural language processing, demonstrating remarkable capabilities in understanding and generating human language Naveed et al. 2023 and to encode common-sense knowledge. They have become advanced artificial intelligence systems capable of understanding y Arcas 2022 , generating, and generalizing text across various tasks Radford et al. 2019 ; Brown et al. 2020 . Recently, LLMs have shown reasoning and inferring capacity Qian et al. 2023 ; Yao et al. 2023 , which improve the agent decision-making ability. While the research community has investigated the ability of LLMs to understand human language, it has overlooked investigating the ability of LLMs to understand agent ”language” which is the internal state representation of an agent during the learning process, and the ability of LLMs in aligning the internal representation of humans and the internal representation of an artificial learning agent of the same environment. In our paper, we focus on the following question: for an emergent symbolic representation in developmental learning agents, can LLMs translate human instructions from natural language into the internal representation of the agent? In order to explore the question, we analyze the performance of LLMs to align human instructions in natural language with the dynamic symbolic representation developed by a reinforcement learning agent. Our work is based on the hierarchical reinforcement learning algorithm Spatio-Temporal Abstraction Via Reachability (STAR) Zadem et al. 2024 , which was shown to learn a symbolic representation of the state space Zadem et al. 2023 grounded on its environments. The main contributions of our paper are: • We show that LLMs can be a tool to translate human natural language instructions into the internal representation of STAR during its learning process. • We show that the capacity of LLMs to align two different representations is limited due to the fact that they do not fully leverage the internal representation learned by the developmental agent. We state the preliminary of our research in section 2 , propose the method in section 3 , and describe the implementation details in section 4 . Then, we design the experiments and analyze the results in section 5 . Finally, we discuss the result and give our conclusion in section 6 and section 7 . Figure 1 : (a)(d) Environments, (b)(e) Average success rate of STAR (from Zadem 2024 ), (c)(f) Partition into regions of STAR, in respectively Ant Maze and Ant Fall. The regions in (c)(f) are the internal representation emerging during the training at timestamps noted in (b) and (e). The red point represents the initial position of the robot while the yellow point represents the goal position. Our LLM-based system translates instructions to guide the robot (such as ”go east to the end, turn north until past the wall and go west until the end”), into a sequence of traversed regions (for Partition II of AntMaze, the output is 5 → 11 → 2 → 3 → 4). 2 Preliminary We base our work on running STAR at Ant Maze Duan et al. 2016 and Ant Fall Nachum et al. 2018 tasks. STAR is a 3-layers Hierarchical Reinforcement Learning algorithm aiming to solve long-horizon tasks: the high-level agent selects regions in an abstract goal space, the middle-level agent selects concrete subgoals that aid in achieving abstract goals, and the low-level agent learns to take action in the environment. STAR automatically discovers discrete symbolic representations of environment by grouping states with similar reachability properties into symbolic goal regions and allowing incremental refinement. The abstract symbols evolve over time as new experiences improve the learned representation. When running STAR on one task, a set of internal symbolic representation data S=\{s_{1},s_{2},...,s_{N}\} emerges. Based on the mechanism of STAR, s_{i} represents a set of emergent internal symbols that align with environment dynamics, and the s_{i+1} is built on the top of s_{i} . We collect the symbolic partitions that emerge during the execution of the STAR algorithm on the Ant Maze and Ant Fall environments, as illustrated in Fig. 1 (a) and (d). The Ant Maze environment features a quadrupedal robotic agent navigating through a “ \supset ”-shaped corridor toward a fixed goal location, and the Ant Fall task simulates an environment where the robot is initially placed on a raised platform separated by a pit from its target. The agent must learn that it needs to

Read the complete text