Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion

Nimesh Khandelwal, Prof. Shakti S. Gupta · 2026

Read Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion by Nimesh Khandelwal, Prof. Shakti S. Gupta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-27416.txt.

Introduction

Khandelwal and Gupta report a case study in AI-led robot research. A human set broad goals through a coding tool. An AI agent ran most of the work loop. It read code, fixed faults, and launched new trials. Over seventy trials ran on a four-legged robot in simulation. Best runs reached low speed error and 97 percent timeout rates. The agent also traced PhysX crashes to certain terrain shapes. The study shows both the power and the limits of agent-led work.

Catalog record

CREATORNimesh Khandelwal, Prof. Shakti S. Gupta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38449
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.27416

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Agent-Driven Autonomous Reinforcement Learning Research: Iterative Policy Improvement for Quadruped Locomotion Nimesh Khandelwal, Prof. Shakti S. Gupta Abstract This paper documents a case study in agent-driven autonomous reinforcement learning research for quadruped locomotion. The setting was not a fully self-starting research system. A human provided high-level directives through an agentic coding environment, while an agent carried out most of the execution loop: reading code, diagnosing failures, editing reward and terrain configurations, launching and monitoring jobs, analyzing intermediate metrics, and proposing the next wave of experiments. Across more than 70 experiments organized into fourteen waves on a DHAV1 12-DoF quadruped in Isaac Lab, the agent progressed from early rough-terrain runs with mean reward around 7 to a best logged Wave 12 run, exp063, with velocity error 0.263 and 97% timeout over 2000 iterations, independently reproduced five times across different GPUs. The archive also records several concrete autonomous research decisions: isolating PhysX deadlocks to terrain sets containing boxes and stair-like primitives, porting four reward terms from openly available reference implementations [ 10 , 9 ] , correcting Isaac Sim import and bootstrapping issues, reducing environment count for diagnosis, terminating hung runs, and pivoting effort away from HIM after repeated terrain=0.0 outcomes. Relative to the AutoResearch paradigm [ 1 ] , this case study operates in a more failure-prone robotics RL setting with multi-GPU experiment management and simulator-specific engineering constraints. The contribution is empirical and documentary: it shows that an agent can materially execute the iterative RL research loop in this domain with limited human intervention, while also making clear where human direction still shaped the agenda. I Introduction Recent interest in autonomous ML research has shifted from code completion toward closed-loop experimentation, where an agent proposes changes, edits code, runs experiments, inspects outputs, and decides what to try next. The AI Scientist demonstrated that an agent can generate, execute, and write up ML experiments end-to-end [ 3 ] , and its successor extended this to workshop-level papers via agentic tree search [ 15 ] . Karpathy’s AutoResearch project is a compact public example of this pattern: an agent iterates on a single-GPU training setup, measures the result, and keeps searching without a human manually editing the training code each time [ 1 ] . Agent Laboratory further explored using agents as research assistants across the full scientific workflow [ 12 ] . That framing raises a practical question for robotics RL: can the same style of agent-driven research work when experiments depend on physics simulation, reward engineering, terrain curricula, simulator bootstrapping, and multi-GPU job management? This paper reports one case study. The underlying task is rough-terrain locomotion for the DHAV1 12-DoF quadruped in Isaac Lab. The novelty is not a new PPO variant. Instead, the contribution is a documented research process in which an agent, operating through an agentic coding environment, executed most of the research loop over fourteen waves of experiments. The human operator set high-level goals such as tuning both Basic PPO and HIM, or consulting openly available reference implementations for rough-terrain settings. The agent then handled the lower-level work: comparing code paths, identifying failure modes, modifying reward terms, launching jobs onto available GPUs, watching for hangs, killing bad runs, and deciding which follow-up experiments were worth the next wave. The claims in this paper are intentionally narrow. We do not claim that agents can autonomously solve robotics RL in general, or that human guidance was absent. In this case study, the human defined the broad direction and the agent executed most of the detailed research actions. Within that scope, the archive shows a measurable improvement from early Wave 1 reward levels around 7 to a best Wave 12 velocity error of 0.263 and 97% episode survival in exp063, with convergence confirmed across two additional waves of experiments that failed to improve upon this result, alongside several engineering discoveries that changed the course of the study. II Related Work II-A Autonomous Research with Agents AutoResearch frames agents as active experimenters rather than passive assistants: the agent edits training code, runs short experiments, evaluates the result, and iterates [ 1 ] . Its public implementation is deliberately compact, centered on a small Python codebase and single-GPU language-model training. The AI Scientist [ 3 ] and its v2 successor [ 15 ] demonstrated that agents can conduct end-to-end ML research including paper generation. Agent Laboratory [ 12 ] studied agents as research assistants across the full scientific workflow. The present work is similar in spirit but differs in operating conditions. Our setting includes physics simulation, terrain generation, runtime initialization issues in Isaac Sim, reward decomposition, and concurrent use of four GPUs. That makes the agent’s job less about pure hyperparameter search and more about mixed research and engineering triage. Coding agents such as SWE-agent [ 16 ] and agentic development environments make this kind of loop increasingly practical because they combine code editing, shell access, file inspection, and iterative execution within one interface. This paper does not evaluate such systems in the abstract. It reports what one such system did in a particular RL project. II-B Reward Design and Robotics RL Infrastructure PPO remains a common baseline for legged locomotion because it is simple to implement and scales well to large batches of parallel simulation [ 13 ] . Rudin et al. demonstrated that massively parallel GPU simulation can train locomotion policies in minutes [ 11 ] , while Miki et al. showed robust perceptive locomotion in the wild using learned policies trained in simulation [ 5 ] . Isaac Sim and Isaac Lab provide the simulation and environment infrastructure used in many modern quadruped training pipelines [ 8 , 7 , 6 ] . In these settings, progress depends not only on the optimizer but also on terrain curricula, reward shaping, reset logic, and simulator stability. On the reward-design side, recent work has shown that agents can generate effective reward functions. Eureka [ 4 ] used agent-generated reward code to match or exceed human-designed rewards on dexterous manipulation and locomotion tasks. Text2Reward [ 14 ] similarly used language models to produce dense reward signals from natural-language descriptions. In our case study, the agent did not generate reward functions from scratch but rather identified, ported, and tuned reward terms from openly available reference code—a complementary mode of agent-assisted reward engineering. Openly available reference implementations also matter. One such setup [ 10 , 9 ] provides a practical quadruped training recipe with terrain simplifications and dense locomotion rewards that informed later waves in this archive. HIMLoco motivated the HIM branch evaluated here [ 2 ] . As in the original archive, we do not claim a general comparison between PPO and HIMLoco. We report only that the specific HIM port used in this project consistently failed to achieve nonzero terrain progression in the final rough-terrain configuration. III System Architecture The research loop combined a human supervisor, an agent, the project repository, and the experiment runtime shown in Fig. 1 conceptually, although no separate figure is needed to follow the paper. The key components were: 1. Human instruction layer : brief high-level directives such as “tune both Basic PPO and HIM” or “consult reference implementations for rough terrain config.” 2. Agent execution layer :

Read the complete text