Endless Terminals: Scaling RL Environments for Terminal Agents

Kanishk Gandhi, Shivam Garg, Noah D. Goodman, Dimitris Papailiopoulos · 2026

Read Endless Terminals: Scaling RL Environments for Terminal Agents by Kanishk Gandhi, Shivam Garg, Noah D. Goodman, Dimitris Papailiopoulos (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-16443.txt.

Introduction

Kanishk Gandhi and three coauthors grow end less shell task worlds for agent train. Their 2026 preprint deals with tools that act in a text shell. Past task sets were built to test agents, not to train them. Train by trial needs a scale path, not a fixed set. The team builds a full auto path that makes new tasks on its own.

Each new task comes with checks that can grade the agent. Fresh tasks stop agents from rote learn of a fixed set. Scale of fresh tasks lets trial learn run long and wide. That breadth is what self growth needs most. The work, named Endless Terminals, feeds that need.

Catalog record

CREATORKanishk Gandhi, Shivam Garg, Noah D. Goodman, Dimitris Papailiopoulos
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-16443

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Endless Terminals: Scaling RL Environments for Terminal Agents Kanishk Gandhi, Shivam Garg, Noah D. Goodman, Dimitris Papailiopoulos Abstract Environments are the bottleneck for self-improving agents. Current terminal benchmarks were built for evaluation, not training; reinforcement learning requires a scalable pipeline, not just a dataset. We introduce Endless Terminals , a fully autonomous pipeline that procedurally generates terminal-use tasks without human annotation. The pipeline has four stages: generating diverse task descriptions, building and validating containerized environments, producing completion tests, and filtering for solvability. From this pipeline we obtain 3255 tasks spanning file operations, log management, data processing, scripting, and database operations. We train agents using vanilla PPO with binary episode level rewards and a minimal interaction loop: no retrieval, multi-agent coordination, or specialized tools. Despite this simplicity, models trained on Endless Terminals show substantial gains: on our held-out dev set, Llama-3.2-3B improves from 4.0% to 18.2%, Qwen2.5-7B from 10.7% to 53.3%, and Qwen3-8B-openthinker-sft from 42.6% to 59.0%. These improvements transfer to human-curated benchmarks: models trained on Endless Terminals show substantial gains on held out human curated benchmarks: on TerminalBench 2.0, Llama-3.2-3B improves from 0.0% to 2.2%, Qwen2.5-7B from 2.2% to 3.4%, and Qwen3-8B-openthinker-sft from 1.1% to 6.7%, in each case outperforming alternative approaches including models with more complex agentic scaffolds. These results demonstrate that simple RL succeeds when environments scale. 1 1 1 Code available at https://github.com/kanishkg/endless-terminals Figure 1: Endless Terminals. Tasks are procedurally generated through four phases: (I) task description generation, (II) container setup with iterative validation, (III) completion test generation, and (IV) solution-based filtering using o3. The pipeline yields 3255 verified tasks for training terminal agents with PPO. 1 Introduction Reinforcement learning is hungry for environments. Successes of RL in improving language model reasoning, from mathematical problem solving to code generation have relied on access to a large set of diverse, and automatically verifiable tasks. Yet for training LLM agents to perform multi-turn computer tasks in a terminal, no such scalable environment exists. Real world terminal use requires agents to reason across multiple interactions, recover from errors, and interactively execute sequences of commands that transform the stat of a system. Manual curation of such environments is expensive, and existing benchmarks offer at most hundreds of tasks, far too few to support robust RL training ( 23 ; 16 ; 22 ) . Prior approaches to building capable terminal agents have largely sidestepped this environment bottleneck. Some rely on fixed evaluation benchmarks repurposed for training, risking overfitting to narrow task distributions. Others distill behavior from stronger proprietary models through supervised finetuning ( 7 ) , inheriting the ceiling of the teacher and requiring expensive API access. A third line of work ( 22 ; 10 ) uses human curated datasets of coding or shell tasks, but the annotation cost limits scale and diversity. What remains missing is a fully autonomous pipeline that can generate an endless stream of terminal tasks: complete with initial environments, task specifications, and verification tests, with minimal human supervision. We introduce Endless Terminals, a procedural generation pipeline that synthesizes terminal-use tasks without human annotation or distillation. The pipeline operates in four stages (see Fig. 2 ): 1) generating diverse task descriptions by sampling across categories, complexity levels, and scenario contexts; 2) building containerized environments and validating them with automatically generated prerequisite tests; 3) generating completion tests that verify the expected end state; and 4) filtering tasks by sampling solutions from a capable model to ensure solvability. We train models with a minimal agent architecture, a simple interaction loop where the model reasons, executes commands, and observes the output every turn, with no retrieval, tool use, or multi-agent scaffolding. Vanilla PPO ( 18 ) with 16 turns on Endless Terminals yields substantial gains: on our held-out dev set, Llama-3.2-3B ( 5 ) improves from 4.0% to 18.2%, Qwen-2.5-7B ( 17 ) from 10.7% to 53.3%, and Qwen-3-8B-openthinker-sft ( 26 ; 22 ) from 42.6% to 59.0%. These gains transfer to difficult, human curated benchmarks. On TerminalBench 2.0 ( 23 ) , Llama-3.2-3B improves from 0.0% to 2.2%, Qwen-2.5-7B from 2.2% to 3.4%, and Qwen-3-8B-Open-Thoughts from 1.1% to 6.7%, in each case outperforming other finetuned versions of the corresponding models on TerminalBench. These results demonstrate that an autonomous pipeline to scale environments can lead to improvements on human curated benchmarks and that simple RL setups succeed if we can scale them up. Figure 2: Overview of the Endless Terminals procedural generation pipeline. Phase I generates diverse task descriptions by sampling across categories, complexity levels, and scenario contexts, producing both a task description and privileged ground truth data for verification. Phase II creates containerized environments and validates them with self written prerequisite tests. Phase III generates completion tests that verify the expected end state. Phase IV filters tasks by sampling 16 solutions from o3, retaining only tasks where at least one solution succeeds. Some example tasks include shell configuration, log analysis, and checksum generation. 2 Related Work Agentic Scaffolds. Prior work develops scaffolds for LLM agents with custom tools, retrieval, context management, and multi-agent coordination. SWE-agent ( 27 ) provides specialized commands for code navigation and editing. OpenHands ( 24 ) combines an agent with tools for bash execution, file editing, and browser interaction. Terminus ( 23 ) takes a simpler approach, giving the agent only an interactive tmux session controlled via keystrokes. Our scaffold is simpler still: the agent reasons and emits commands, with the full history of prior thoughts, actions and shell outputs in context. Supervised Finetuning and Distillation One approach to building capable agents is to curate high quality datasets for supervised finetuning. 7 (OpenThoughts) distill reasoning traces from frontier models, showing that data quality drives downstream improvement with RL. 4 identify cognitive behaviors: verification, backtracking, subgoal setting that enable self-improvement via RL, and show these can be instilled. 11 demonstrate how midtraining on targeted data mixes to elicit capabilities like math and coding aids improvement with RL. Finally, OpenThinker-Agent ( 22 ) applies this recipe to terminal tasks, distilling traces from strong teachers. These approaches are complementary: SFT can provide a warm start for RL training. Benchmarks and Interactive Environments. Progress on coding agents has been driven by human curated benchmarks with execution based evaluation, including SWEBench ( 8 ) for GitHub issue resolution and TerminalBench 2.0 ( 23 ) for terminal tasks. These benchmarks require multiturn interaction where agents issue commands and observe outputs before deciding the next action, a format shared by InterCode ( 28 ) for interactive coding, and WebArena ( 32 ) for web navigation. Our procedurally generated tasks follow this multiturn format, with agents interacting with a persistent shell. Synthetic Environment Generation Some recent works construct environments for training agents on verifiable tasks. SWEGym ( 13 ) provides 2438 Python tasks with executable tests, but relies on existing GitHub issues rather than procedural generation. In single turn domains, 15 propose models that posit and solve increasingly difficult problems training them through self-play. OpenThoughts Agent ( 22 ) is closest to our work, introducing terminal-use tasks for SFT and RL. However, their RL dataset involves human generated queries and commands from NL2Bash ( 10 ) , and doesn’t yield gains on benchmarks like Terminal Bench 2.0, or an autonomously generated endless

Read the complete text