A R M A P : Scaling autonomous agents via A utomatic R eward M odeling A nd P lanning
Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, Chuang Gan · 2025
Read A R M A P : Scaling autonomous agents via A utomatic R eward M odeling A nd P lanning by Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, Chuang Gan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-12130.txt.
Introduction
Five authors propose ARMAP, a framework that lets language model agents learn from experience without human labels. Agents still struggle with tasks that need many steps and feedback from the world, such as online shopping or math problems. Gathering training data for such tasks is costly, and many strong models sit behind APIs that block fine-tuning. ARMAP works around both problems by learning a reward model straight from the environment.
First, one agent wanders through a task space at random, producing varied action traces. A second model then assigns each trace a task intent and writes both a correct and a flawed response. These triples train a reward model that scores new traces and guides planning. Tests on several agent benchmarks show the scores generalize across tasks.
Catalog record
| CREATOR | Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, Chuang Gan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2502-12130
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A R M A P : Scaling autonomous agents via A utomatic R eward M odeling A nd P lanning Zhenfang Chen, Delin Chen, Rui Sun, Wenjun Liu, Chuang Gan Abstract Large language models (LLMs) have demonstrated remarkable capabilities across a range of text-generation tasks. However, LLMs still struggle with problems requiring multi-step decision-making and environmental feedback, such as online shopping, scientific reasoning, and mathematical problem-solving. Unlike pure text data, collecting large-scale decision-making data is challenging. Moreover, many powerful LLMs are only accessible through APIs, which hinders their fine-tuning for agent tasks due to cost and complexity. To address LLM agents’ limitations, we propose a framework that can automatically learn a reward model from the environment without human annotations. This model can be used to evaluate the action trajectories of LLM agents and provide heuristics for task planning. Specifically, our approach involves employing one LLM-based agent to navigate an environment randomly, generating diverse action trajectories. Subsequently, a separate LLM is leveraged to assign a task intent and synthesize a negative response alongside the correct response for each trajectory. These triplets (task intent, positive response, and negative response) are then utilized as training data to optimize a reward model capable of scoring action trajectories. This reward model can be integrated with LLM-based agents and various planning algorithms to enhance task-solving performance. The effectiveness and generalizability of our framework are demonstrated through evaluations conducted on different agent benchmarks. In conclusion, our proposed framework represents a significant advancement in enhancing LLM agents’ decision-making capabilities. By automating the learning of reward models, we overcome the challenges of data scarcity and API limitations, potentially revolutionizing the application of LLMs in complex and interactive environments. This research paves the way for more sophisticated AI agents capable of tackling a wide range of real-world problems requiring multi-step decision-making. 1 1 1 Project page: https://armap-agent.github.io 1 Introduction Developing AI agents capable of perceiving environments, understanding instructions, and acting to accomplish a wide range of tasks in interactive settings ( Brooks, 1986 ) have many real-world applications, including virtual human assistants ( Reed et al., 2022 ; Casheekar et al., 2024 ) , business process management ( Kirchdorfer et al., 2024 ) , and robotic process automation ( Rana et al., 2023 ; Ahn et al., 2022 ; Palo et al., 2023 ) . The recent advent of large generative models has revolutionized numerous applications, such as question answering ( Rajpurkar et al., 2016 ) , text summarization ( Hermann et al., 2015 ) , and multi-modal understanding ( Chen et al., 2015 ; Goyal et al., 2017 ; Yu et al., 2016 ) . However, while these models excel in text comprehension and generation tasks, their performance in decision-making scenarios—such as online shopping and scientific reasoning falls relative short of human capabilities. This disparity likely stems from the nature of the training data. Large generative models are typically pre-trained on readily available image and text corpora from the internet. In contrast, trajectory data for agent tasks, which require multi-step interaction with the environment, is more challenging to collect and does not naturally occur on the internet. Furthermore, current state-of-the-art commercial Language Learning Models (LLMs), such as GPT-4V ( OpenAI et al., 2024 ) and Gemini ( Reid et al., 2024 ) , often provide only limited APIs for general users. This restriction renders it either infeasible or cost-prohibitive to fine-tune these models for specific agent tasks, further impeding progress in this field. Figure 1: In Fig. 1 (a), we show that it is difficult for LLM agents to generate multi-step plans in an interactive environment to achieve the instruction goal. However, it is relatively easy for an LLM to learn a reward model that can evaluate whether the trajectories meet the task instructions, as shown in Fig. 1 (b). In Fig. 1 (c), we show that a learned reward model can be used to guide the default policy models to improve action planning. Previous studies have explored the development of autonomous agents for decision-making tasks using large language models (LLMs). Early research ( Yao et al., 2023a ; Zheng et al., 2024 ; Deng et al., 2024 ) utilized instruction prompts with few-shot examples to direct LLMs in handling various agent tasks. These methods do not require task-specific fine-tuning but have shown limited performance on benchmarks requiring interaction with environments and precise action prediction. A different research direction involves collecting human preference data ( Hong et al., 2023 ) or distilling trajectory data from advanced commercial LLM APIs ( Zeng et al., 2023 ; Deng et al., 2024 ) and fine-tuning smaller open-source LLMs to create new policy models for agent tasks. However, this distillation process relies on advanced pre-trained agent models for trajectory data extraction, which are often unavailable, expensive, or subject to commercial restrictions. For instance, data from models such as GPT-4 or Gemini cannot be used for commercial purposes. A fundamental premise of our approach is that, in most agent applications, evaluation is easier than generation ( Karp, 1975 ; Naor, 1996 ) . As illustrated in Fig. 1 (a), generating a correct multi-step solution to navigate to the target page is challenging since it needs to predict multiple actions and interact with the environment. However, it is relatively simple to evaluate whether the output action trajectories and environment states meet the provided intent to find a "vans sport canvas fashion sneaker" . Building on this premise, we suggest that developing a reward model is more feasible than creating a policy model for agent tasks. With an effective reward model, it becomes possible to guide LLMs in planning tasks both effectively and efficiently. For instance, as depicted in Fig. 1 (c), by integrating the reward model with an LLM-based agent and the Monte Carlo Tree Search (MCTS) algorithm ( Silver et al., 2017 ; Coulom, 2006 ) , we can simulate and evaluate the future states of agent tasks, thereby making better decisions for subsequent actions. This approach is analogous to mental simulation ( Hegarty, 2004 ; Lake et al., 2017 ) in cognitive science, where humans envision the outcomes of potential actions to make better decisions in problem-solving. While reward models can assist LLM agents in planning, developing these reward models presents significant challenges. Some prior studies have utilized powerful commercial LLM APIs as evaluators for tasks ( Kwon et al., 2023a ) . Although these approaches have demonstrated effectiveness in certain applications, they rely on state-of-the-art LLM models for evaluation, which are often expensive and difficult to scale. In this paper, we introduce an automated method to learn multi-modal reward models without relying on state-of-the-art LLMs for guidance. Furthermore, previous work has not considered integrating the learned reward models with various planning algorithms for problem-solving. The process of learning the reward model involves three steps. Initially, we utilize an LLM-based agent (e.g., Dubey et al. (2024) ) to navigate in the environments, aiming to achieve a randomly proposed intent while collecting extensive action trajectory demonstrations. Subsequently, the LLM model examines the collected trajectories and proposes a refined intent that the sampled trajectories actually accomplish. Additionally, we prompt the LLM to generate negative trajectories that fail to achieve the intended task. Finally, based on