Language-based Trial and Error Falls Behind in the Era of Experience
Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao · 2026
Read Language-based Trial and Error Falls Behind in the Era of Experience by Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-21754.txt.
Introduction
Haoyu Wang and nine co-authors argue that language-based trial and error falls behind learning from experience. Language models excel at tasks framed in words. On unseen tasks that are symbolic or spatial rather than verbal, they lag badly.
Earlier work blamed the mismatch between training text and the test setting. The authors press further, contrasting verbal feedback loops with learning from lived experience in the environment. Their claim favors agents that improve by acting and observing, not by reading critiques alone. The distinction shapes how the next generation of agents should be built and trained.
Catalog record
| CREATOR | Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 65 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-21754
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Language-based Trial and Error Falls Behind in the Era of Experience Haoyu Wang, Guozheng Ma, Shugang Cui, Yilun Kong, Haotian Luo, Li Shen, Mengya Gao, Yichao Wu, Xiaogang Wang, Dacheng Tao Abstract While Large Language Models (LLMs) excel in language-based agentic tasks, their applicability to unseen, nonlinguistic environments (e.g., symbolic or spatial tasks) remains limited. Previous work ( 4 ) attributes this performance gap to the mismatch between the pretraining distribution and the testing distribution. In this work, we demonstrate the primary bottleneck is the prohibitive cost of exploration: mastering these tasks requires extensive trial-and-error, which is computationally unsustainable for parameter-heavy LLMs operating in a high dimensional semantic space. To address this, we propose SCOUT ( S ub- S cale C ollaboration O n U nseen T asks), a novel framework that decouples exploration from exploitation. We employ lightweight "scouts" (e.g., small MLPs) to probe environmental dynamics at a speed and scale far exceeding LLMs. The collected trajectories are utilized to bootstrap the LLM via Supervised Fine-Tuning (SFT), followed by multi-turn Reinforcement Learning (RL) to activate its latent world knowledge. Empirically, SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming proprietary models, including Gemini-2.5-Pro (0.60), while saving about 60% GPU hours consumption. The code is available at https://github.com/Harry-mic/SCOUT . 1 Introduction Figure 1 : Exploration and Distillation Stage will either directly award the language models the skills, such as the performance on Sokoban-Box2 (below) that leads to direct convergence (0.0 to 0.45), or indirectly teach relevant knowledge that will be later activated via Evolving Stage, such as the performance on Sudoku (above figure, from 0.0, to 0.29, then to 0.97). Figure 2 : Overview of the SCOUT framework . The pipeline consists of three stages: (1) Exploration Stage: Lightweight scouts efficiently capture environmental dynamics to generate expert trajectories; (2) Distillation Stage: These trajectories are textualized to "warm-up" the LLM via supervised fine-tuning; (3) Evolving Stage: The LLM further refines its reasoning and decision making capabilities through multi-turn PPO. Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks ( 9 ; 34 ; 43 ; 32 ; 38 ; 27 ) , primarily driven by extensive pretraining on high quality text corpora ( 3 ; 29 ; 37 ; 46 ) . This equips LLMs with broad world knowledge, enabling strong zero-shot generalization in language correlated scenarios such as creative writing, summarization, reasoning, and even language based agentic tasks ( 15 ; 44 ; 26 ; 27 ; 38 ) . However, when deployed in unseen, non-linguistic tasks such as spatial tasks ( 6 ; 8 ) , symbolic tasks ( 2 ) and complex long horizon tasks ( 19 ; 33 ) , existing pretraining is far from sufficient. These tasks demonstrate that the real world is unbounded, involving "endless complexity" ( 28 ) . Therefore it is hard to fully simplify and cover all tasks during pretraining. LLMs’ rich pretrained knowledge struggles in these scenarios because now they need to internalize the environmental dynamics from scratch rather than directly utilizing the pretrained world knowledge. SPA ( 4 ) attributes this performance gap to the fact that LLMs are significantly less familiar with symbolic state-based tasks ( 2 ) compared to the language based state tasks like Webshop ( 38 ) ,ALFWorld ( 27 ) . SPA separates the in-distribution and out-of-distribution tasks by the state perplexity against random guess. In this work, we focus on these out-of-distribution tasks that are often composed of various symbols or numbers, rather than natural language, and are more alien to language agents. The newly introduced tasks in this work are also OOD tasks, and we show the higher state perplexity against random guess as evidence of OOD tasks as SPA ( 4 ) does in Table 5 . We mainly focus on these symbolic and spatial tasks, and call them "unseen tasks" in this work. Beyond the pretraining stage, the inefficiency of LLM agents in mastering these new tasks also stems from two fundamental mismatches. First , there is a mismatch between the action space and the generation space. Generating a token requires a forward pass through billions of parameters of LLMs, resulting in low efficiency for both exploration and exploitation. Furthermore, an LLM is exploring a vast vocabulary space (typically exceeding 30,000 tokens), whereas many reasoning and symbolic tasks ( 2 ) require only a discrete, low dimensional set of actions. Although settings like temperature and top-k can reduce the candidate tokens, forcing an LLM to search for optimal policies within such a high-dimensional semantic space is computationally wasteful and hinders efficient exploration. Second , depending solely on language priors limits scalability. The "Bitter Lesson" ( 28 ) teaches us that leveraging computation (searching and learning) is far more effective than relying on predefined knowledge in the long run. Although LLMs are semantically rich, they struggle to grasp the specific dynamics of the physical world ( 6 ; 31 ; 36 ) that cannot be fully encoded in text. To bridge this gap, we propose SCOUT ( S ub- S cale C ollaboration O n U nseen T asks), a novel agent framework that harmonizes the exploitation on world knowledge of LLM agents with the exploration efficiency of "scouts". As shown in Figure 2 , our key insight is to decouple the heavy exploration phase of LLM agents from the exploitation phase. Specifically, we employ lightweight neural networks (e.g., small MLPs or CNNs) to serve as "scouts" . Their low parameter count and high inference speed enable them to evolve rapidly with classic Reinforcement Learning (RL) algorithms (e.g., DQN, PPO) to master the environmental dynamics and generate high quality expert trajectories. These trajectories then serve as a warm-up for the LLM. This process effectively distills the specific task dynamics captured by the scouts into the LLMs, activating the LLMs’ internal relevant world knowledge with the specific unseen task. We further conduct multi-turn RL on the LLMs to align them with new tasks. This allows the LLMs to skip the heavy and inefficient exploration phase and focus on the exploitation of newly learned knowledge. We test our methods on several symbolic and dynamic worlds such as FrozenLake, Sokoban and Sudoku. We further investigate the long horizon and spatial ability, and respectively introduce: 1) 2048, a grid game which needs above 800 turns to reach the 2048 tile; 2) Rubiks’ Cube, a game of restoring a scrambled Rubiks’ Cube. Empirical results demonstrate SCOUT significantly outperforms baselines. SCOUT enables a Qwen2.5-3B-Instruct model to achieve an average score of 0.86, significantly outperforming the tested baselines and proprietary models, within which Gemini-2.5-Pro achieves the highest 0.60 score. To summarize, we propose a novel framework, SCOUT. By leveraging small neural networks as scouts for rapid exploration and trajectory generation, we alleviate the exploration efficiency bottleneck inherent in pure LLM agents and activate the learned world knowledge with multi-turn RL. We demonstrate that SCOUT effectively activates the LLM’s potential for unseen OOD tasks, allowing it to model the new environment efficiently and effectively. 2 Sub-Scale Collaboration On Unseen Tasks In this section, we introduce the agentic framework SCOUT (Sub-Scale Collaboration On Unseen Tasks) from four aspects. First, we explain