LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search
Masakazu Yoshimura, Zitang Sun, Yuiko Sakuma, Junji Otsuka, Atsushi Irie, Takeshi Ohashi · 2026
Read LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search by Masakazu Yoshimura, Zitang Sun, Yuiko Sakuma, Junji Otsuka, Atsushi Irie, Takeshi Ohashi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16555.txt.
Introduction
Yoshimura and colleagues propose LLMasTool, a tree frame for neural design search. Code search relies on hand-crafted spaces that limit open work. Language-model methods reduce manual design but struggle to make designs and show training-data biases. LLMasTool pulls modules from source code and represents designs as layered trees. Growth proceeds through tree changes, not code writing, with diverse Bayesian planning at coarse level. The model settles degrees of freedom, giving a working design and a path. It beats methods by 0.69, 1.83 and 2.68 on CIFAR-10, CIFAR-100 and ImageNet16-120.
Catalog record
| CREATOR | Masakazu Yoshimura, Zitang Sun, Yuiko Sakuma, Junji Otsuka, Atsushi Irie, Takeshi Ohashi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112704 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.16555
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM as a Tool, Not an Agent: Code-Mined Tree Transformations for Neural Architecture Search Masakazu Yoshimura, Zitang Sun, Yuiko Sakuma, Junji Otsuka, Atsushi Irie, Takeshi Ohashi Abstract Neural Architecture Search (NAS) aims to automatically discover high-performing deep neural network (DNN) architectures. However, conventional algorithm-driven NAS relies on carefully hand-crafted search spaces to ensure executability, which restricts open-ended exploration. Recent coding-based agentic approaches using large language models (LLMs) reduce manual design, but current LLMs struggle to reliably generate complex, valid architectures, and their proposals are often biased toward a narrow set of patterns observed in their training data. To bridge reliable algorithmic search with powerful LLM assistance, we propose LLMasTool, a hierarchical tree-based NAS framework for stable and open-ended model evolution. Our method automatically extracts reusable modules from arbitrary source code and represents full architectures as hierarchical trees, enabling evolution through reliable tree transformations rather than code generation. At each evolution step, coarse-level planning is governed by a diversity-guided algorithm that leverages Bayesian modeling to improve exploration efficiency, while the LLM resolves the remaining degrees of freedom to ensure a meaningful evolutionary trajectory and an executable generated architecture. With this formulation, instead of fully agentic LLM approaches, our method explores diverse directions beyond the inherent biases in the LLM. Our method improves over existing NAS methods by 0.69, 1.83, and 2.68 points on CIFAR-10, CIFAR-100, and ImageNet16-120, demonstrating its effectiveness. 1 Introduction Figure 1 : Overview of the proposed NAS pipeline. (Top-left) we mine reusable modules from source code to build a module database. (Top-right) architectures are represented as deploy-friendly hierarchical trees. (Bottom-left) an evolution engine applies controlled tree transformations using LLM as a tool for fine-grained refinement. Designing high-performing neural architectures remains a key driver of progress in deep learning research, yet it often demands substantial expertise and extensive trial-and-error. Neural Architecture Search (NAS) aims to automate this process by exploring candidate architectures and optimizing validation performance. Early NAS demonstrated that reinforcement-learning [ 76 ] and evolutionary strategies [ 47 ] can discover competitive architectures. Later work showed that searching for transferable “cells” and reusing them can reduce manual engineering while maintaining strong performance [ 77 ] . NAS has since become more practical through improved efficiency. One-shot or parameter-sharing methods amortize training across many candidates [ 42 ] , differentiable NAS enables gradient-based optimization over relaxed architectural choices [ 31 ] , and surrogate-guided approaches improve sample efficiency by learning predictors to prioritize evaluations [ 30 ] . However, most of them rely on handcrafted search spaces or primitives to guarantee executability, which restricts open-ended exploration and biases discoveries toward designers’ priors [ 15 ] . Another line of work broadens exploration by evolving architectures via structured edits to existing networks. Function-preserving transformations such as network morphism reduce training cost during iterative editing [ 56 ] , and Auto-Keras further couples such edits with Bayesian optimization based on historical outcomes [ 24 ] . Hierarchical decompositions factorize the search across abstraction levels [ 29 , 44 ] . Yet, as edit spaces become highly granular across module choices, connectivity patterns, and hyperparameter, the effective branching factor grows combinatorially, making feedback sparse at the action level [ 24 , 48 ] . More recently, large language models (LLMs) have been explored on NAS. The promising merit is to reduce manual search-space engineering by proposing or refining architectures through natural-language interaction. Beyond providing candidate edits with huge search space, LLMs are capable of abstracting the enormous low-level action space into a smaller set of semantically meaningful decision templates, thereby reducing the effective decision complexity while leveraging strong intelligence for plausible module combinations and hyperparameter settings [ 4 , 73 ] . Nonetheless, LLM-driven NAS often suffer from limited controllability and difficulty guaranteeing executability for complex designs, especially when the LLM is used as an autonomous agent for end-to-end generation [ 63 ] . Moreover, LLM proposals tend to be biased toward recurring patterns in their training corpus, which hinder truly open-ended exploration. These observations motivate a different stance: rather than treating the LLM as a fully autonomous architecture generator, we use it as a tool that assists localized decisions inside an algorithmically constrained, executable framework. In this work, we propose LLMasTool, a NAS framework that bridges reliable algorithmic evolution with LLM assistance for stable and open-ended model evolution. Rather than naively merging the two paradigms, we integrate them through a deploy-friendly hierarchical tree representation of DNN architectures. Most of the aforementioned algorithm-based NAS methods [ 76 , 47 , 77 , 42 , 31 , 30 ] search over data-flow graphs, where intermediate features are nodes and operations are edges. While expressive, this fine-grained representation makes it difficult to leverage knowledge embedded in existing source code, which serves as a natural medium for LLMs. In common DNN frameworks such as PyTorch [ 43 ] , models are defined as hierarchically nested modules, with data-flow specified separately in each module. Mapping such implementations into data-flow graphs is non-trivial and often requires intrusive code rewriting. To address this, we represent each architecture as a hierarchical tree, spanning macro-level modules, components, and leaf-level hyperparameters. By automatically analyzing arbitrary source code, LLMasTool constructs a reusable module database that enables direct reuse of existing implementations and knowledge assets. Architectures are evolved via controlled tree transformations rather than free-form code generation, allowing complex yet executable designs. To further support open-ended exploration beyond the LLM’s prior knowledge, we avoid a fully agentic paradigm. Instead, we adopt a hierarchical coarse-to-fine decision framework in which the LLM functions as a structured tool. Coarse-level planning is performed by a diversity-guided evolutionary algorithm, while fine-grained decisions are resolved by the LLM to ensure meaningful and executable transformations. To improve efficiency, historical outcomes are accumulated at an upper layer, where Bayesian modeling guides exploitation within the open-ended search space. Experiments on CIFAR-10, CIFAR-100, ImageNet-100, and ImageNet-1K demonstrate the effectiveness of our several proposals. The contributions of this work are summarized as follows: • We introduce LLMasTool, a hybrid NAS framework that represents full architectures as a hierarchical tree, enabling precise and executable edits while facilitating direct incorporation of prior knowledge. • We design a coarse-to-fine framework that combines diversity-guided global planning with LLM-assisted fine-grained edits, enabling controllable, open-ended exploration beyond the LLM’s priors. • LLMasTool achieves state-of-the-art results, improving performance by 0.69, 1.83, and 2.68 points on CIFAR-10, CIFAR-100, and ImageNet16-120. 2 Related Work 2.1 Neural Architecture Search (NAS) NAS optimizes neural network architectures in a data-driven manner. Most methods represent the architectures as data-flow graphs, where layers or blocks of DNN are nodes and their input-output relationships are edges, optimizing node types, hyperparameters, and connectivity. Since graph manipulation is non-differentiable, reinforcement learning [ 76 , 77 , 42 ] and evolutionary algorithms [ 54 , 30 , 52 ] are widely used to develop the graph. Methods that extract preferable sub-networks from pre-defined super-networks have also been actively studied, including differentiable edge selection [ 31 , 58 , 3 , 53 , 68 ] and sampling partial weights and blocks [ 67 , 2 , 71 , 34 , 23 , 40 , 5 , 6 , 49 ] . Recently, GNN-based methods have been proposed for efficient optimization [ 37 ,