ArchPilot : A Proxy-Guided Multi-Agent Approach for Machine Learning Engineering

Zhuowen Yuan, Tao Liu, Yang Yang, Yang Wang, Feng Qi, Kaushik Rangadurai, Bo Li, Shuang Yang · 2025

Read ArchPilot : A Proxy-Guided Multi-Agent Approach for Machine Learning Engineering by Zhuowen Yuan, Tao Liu, Yang Yang, Yang Wang, Feng Qi, Kaushik Rangadurai, Bo Li, Shuang Yang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-03985.txt.

Introduction

Zhuowen Yuan is the lead author among eight authors on ArchPilot, a multi-agent system for automated machine learning engineering that avoids judging every candidate with a full training run. Full runs create heavy cost, slow cycles, and poor scaling to large search spaces, the team note.

Three agents divide the work. An orchestration agent runs a tree-search-inspired process with restarts and memory. A generation agent proposes and debugs architectures. An evaluation agent runs cheap proxy training and combines scores into a fidelity-aware metric. On MLE-Bench, ArchPilot beat strong baselines named in the work. Proxy guidance lets limited budgets focus on promising candidates first.

Catalog record

CREATORZhuowen Yuan, Tao Liu, Yang Yang, Yang Wang, Feng Qi, Kaushik Rangadurai, Bo Li, Shuang Yang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-03985

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ArchPilot : A Proxy-Guided Multi-Agent Approach for Machine Learning Engineering Zhuowen Yuan, Tao Liu, Yang Yang, Yang Wang, Feng Qi, Kaushik Rangadurai, Bo Li, Shuang Yang Abstract Recent LLM-based agents have demonstrated strong capabilities in automated ML engineering. However, they heavily rely on repeated full training runs to evaluate candidate solutions, resulting in significant computational overhead, limited scalability to large search spaces, and slow iteration cycles. To address these challenges, we introduce ArchPilot , a multi-agent system that integrates architecture generation, proxy-based evaluation, and adaptive search into a unified framework. ArchPilot consists of three specialized agents: an orchestration agent that coordinates the search process using a Monte Carlo Tree Search (MCTS)-inspired novel algorithm with a restart mechanism and manages memory of previous candidates; a generation agent that iteratively generates, improves, and debugs candidate architectures; and an evaluation agent that executes proxy training runs, generates and optimizes proxy functions, and aggregates the proxy scores into a fidelity-aware performance metric. This multi-agent collaboration allows ArchPilot to prioritize high-potential candidates with minimal reliance on expensive full training runs, facilitating efficient ML engineering under limited budgets. Experiments on MLE-Bench demonstrate that ArchPilot outperforms SOTA baselines such as AIDE and ML-Master, validating the effectiveness of our multi-agent system. † † affiliation: Meta Ranking AI Research † † affiliation: UIUC † † contribution: Work done during an internship at Meta † † date: August 10, 2026 † † correspondence: [email protected] , [email protected] 1 Introduction Automating the design of machine learning (ML) pipelines has long been a central objective in AutoML and neural architecture search (NAS), offering the potential to liberate practitioners from the laborious task of manually tuning architectures, training procedures, and optimizing hyperparameters ( He et al. 2021 ) . Despite impressive progress in neural architecture search, hyperparameter optimization, and end-to-end AutoML systems, most prior work still operates on constrained optimization space ( Ying et al. 2019 ; Dong and Yang 2020 ) , which severely limits scalability in real-world settings. Recently, large language models (LLMs) have demonstrated remarkable capability in code generation, reasoning, and multi-step problem solving. This has led to the emergence of LLM-based ML agents that attempt to autonomously design, debug, and iteratively improve ML pipelines. Early systems such as OpenHands ( Wang et al. 2024 ) and MLAgentBench ( Huang et al. 2023 ) established benchmarks and toolchains for evaluating ML agents, while more sophisticated approaches such as AIDE ( Jiang et al. 2025 ) , R&D-Agent ( Yang et al. 2025 ) , and ML-Master ( Liu et al. 2025 ) have shown that LLMs can iteratively refine code by combining exploration and reasoning over a tree of candidate solutions. These systems typically generate runnable scripts, execute them to obtain feedback, and use that feedback to improve subsequent generations. However, a major limitation remains: candidate evaluation is still dominated by repeated full training runs, which are computationally expensive and make it infeasible to explore the vast solution space under realistic compute budgets. As a result, these systems either evaluate very few candidates—hurting coverage and diversity—or consume prohibitively large amounts of compute, reducing their practicality in production and research environments. In this work, we introduce ArchPilot , a multi-agent framework for cost-efficient NAS that explicitly decouples generation, evaluation, and orchestration into three collaborating agents. Unlike prior systems that couple all functionality into a single LLM loop, ArchPilot employs: • an Orchestration Agent (OA) that maintains a Monte Carlo Tree Search (MCTS)-style search tree, selects nodes for expansion using an Upper Confidence Bound for Trees (UCT) criterion ( Kocsis and Szepesvári 2006 ) , backpropagates rewards, and manages structured memory to provide context to GA while preventing redundant exploration, • a Generation Agent (GA) that produces initial drafts, repairs failing pipelines, and proposes atomic, testable improvements conditioned on task description and memory, and • an Evaluation Agent (EA) that instruments and launches proxy-training or full-training scripts, computes multiple proxy signals, adaptively reweights them via ridge-regularized least squares with hard-zero constraints, and maintains a calibrated mapping from proxies to true scores This separation of concerns allows each agent to focus on its specialty: GA maximizes diversity and code quality, EA provides principled and adaptive evaluation, and OA allocates compute and ensures global search consistency. A key novelty of ArchPilot is its multi-proxy evaluation with adaptive reweighting . Rather than relying on a single heuristic or costly full training, EA evaluates each candidate using a small set of cheap but complementary proxies—such as one-epoch validation, noisy validation, and feature-dropout validation ( Li et al. 2020 ; Ha et al. 2023 ) —to capture generalization, robustness, and feature reliance. It then forms a direction-aligned weighted sum of these signals to obtain a node value for tree search. As labeled tuples (y,{\bm{x}}) accumulate from full-training runs, EA refits the proxy weights and OA triggers a tree restart , rescoring and reseeding from the top- k verified nodes to keep exploration aligned with the most up-to-date evaluation signals. Overall, ArchPilot turns NAS into a closed-loop, reasoning-driven process that balances efficiency and accuracy: GA continuously proposes diverse candidates, EA provides fast yet adaptive evaluation, and OA strategically allocates expensive full-training budget where it is most informative. This design enables ArchPilot to explore a significantly larger portion of the search space under the same compute budget compared to prior LLM-based systems, while maintaining reproducibility and interpretability through structured memory and node-level statistics. Our contributions can be summarized as follows: • We present ArchPilot , a three-agent NAS framework that cleanly separates code generation, proxy-based evaluation, and tree-based orchestration, enabling modular upgrades and fine-grained control over search dynamics. • We introduce a principled multi-proxy evaluation pipeline with direction alignment, normalized aggregation, and ridge-regularized weight fitting under a hard-zero policy, yielding reliable value estimates even under noisy or partially missing signals. • We develop a restart-enabled MCTS search algorithm that refreshes exploration whenever the scoring semantics change, preventing stale statistics from misleading the search and focusing compute on empirically strong regions. We evaluate ArchPilot on MLE-Bench ( Chan et al. 2024 ) , demonstrating consistent improvements over strong baselines such as AIDE ( Jiang et al. 2025 ) and ML-Master ( Liu et al. 2025 ) . For example, ArchPilot surpasses 66% of teams on average on MLE-Bench Lite, while the number is 60% for ML-Master and only 55% for AIDE. Our results show that adaptive proxy optimization and restart-enabled exploration together yield higher true performance at lower compute cost, underscoring the promise of multi-agent and reasoning-driven NAS. 2 Related Work 2.1 Traditional NAS Neural architecture search (NAS) aims to automatically discover high-performing neural networks, but early approaches were prohibitively expensive because each candidate architecture required full training and evaluation ( He et al. 2021 ) . This motivated a variety of strategies to approximate candidate performance more efficiently. One common approach is to restrict the search space and apply surrogate models, e.g., early-stopping, weight-sharing, or training on reduced datasets ( Pham et al. 2018 ; Liu et al. 2018 ) . EcoNAS and related methods further improved efficiency by adaptively determining early-stopping thresholds to terminate poor candidates sooner ( Zhou et al. 2020 ) . Although effective, these approaches still require gradient updates for each candidate, limiting scalability under tight

Read the complete text