Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search
Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl, Zitang Sun Junji Otsuka Atsushi Irie Takeshi Ohashi · 2026
Read Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search by Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl, Zitang Sun Junji Otsuka Atsushi Irie Takeshi Ohashi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-19247.txt.
Introduction
Yuiko Sakuma and colleagues rethink neural architecture search, whose fixed search spaces often limit discovery. Language-model helpers can open the space up, yet they tend to wander through biased or weak design ideas. The team first structures design knowledge into a template, which a language model fills by reading papers, then explores it with FairNAD, a mutation scheme that samples ideas fairly and tracks trade-offs. The search finds architectures beating current leaders on CIFAR-10, CIFAR-100, and a 120-class ImageNet variant.
Catalog record
| CREATOR | Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl, Zitang Sun Junji Otsuka Atsushi Irie Takeshi Ohashi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 112135 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.19247
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Structuring Open-Ended NAS: Semi-Automated Design Knowledge Structuring with LLMs for Efficient Neural Architecture Search Yuiko Sakuma, Masakazu Yoshimura, Marcel Gröpl, Zitang Sun Junji Otsuka Atsushi Irie Takeshi Ohashi Abstract Current neural architecture search (NAS) methods are often limited by their predefined, restrictive search spaces. While recent large language model (LLM)-assisted NAS methods enable open-ended search spaces, they often suffer from inefficient exploration due to biased or low-quality design ideas. To address these issues, we propose to semi-automatically structure model design knowledge to guide the search process. Our approach first defines a high-level structural template of architectural attributes. An LLM then populates this template by analyzing papers, creating a rich and diverse search space that embodies this structured design knowledge. To efficiently explore this vast space, we introduce FairNAD, using a multi-type mutation that enables broad exploration through mutation with fair idea sampling, Pareto-aware mutation, LLM-driven iterative mutation, and a fine-grained feedback loop. We demonstrate the effectiveness of FairNAD in discovering high-performing architectures that yield 0.84, 2.17, and 2.35 points improvement on CIFAR-10, CIFAR-100, and ImageNet16-120, respectively, compared to current state-of-the-art methods. 1 Introduction Designing high-performance neural network architectures remains a challenge, as manual architecture design heavily relies on an extensive trial-and-error process conducted by experts. This process is not only time-consuming but also confines the search to the scope of the designer’s expertise, which limits the performance improvements of the resulting models. To address these issues, Neural Architecture Search (NAS) has been extensively investigated as an approach to automate the design process, aiming to reduce search costs while enabling a broader search [ 27 , 61 ] . However, many conventional NAS methods depend on a closed search space, predefined by human experts. This means the search is restricted to combinations of existing operators and blocks, fundamentally limiting the ability to generate innovative architectures. In recent years, a new paradigm has emerged to overcome this limitation: leveraging Large Language Models (LLMs) for NAS. LLM-based NAS adopts an approach of directly generating the source code of a neural network model, either entirely or partially [ 5 , 35 , 59 , 54 ] . This enables an “open-ended” search, removing the limitations imposed by the predefined search space in traditional methods. As a result, LLM-based NAS facilitates the discovery of novel operations, modules, and network topologies, potentially enabling the design of high-performing architectures. Moreover, recent studies [ 54 ] have shown that external design knowledge, such as architectural knowledge extracted from papers, can further enrich this open-ended search process. However, open-endedness alone does not guarantee efficient exploration. The effectiveness of knowledge-guided open-ended NAS critically depends on how the underlying design knowledge is collected, organized, and sampled. In existing methods, design knowledge is directly obtained from LLM priors [ 59 ] , network architectures [ 60 ] , papers [ 54 ] , or web resources [ 9 , 56 ] , and is used in a relatively flat and unstructured manner. These methods retrieve and utilize design knowledge without considering its scope of application (e.g., whether it pertains to an operation, connectivity or the entire network) and its attribute coverage (e.g., neural operators may fall into categories such as convolutions, Transformers, multi-layer perceptrons, and pooling). As a result, knowledge with different levels of abstraction, quality, and applicability are mixed together, making it difficult to effectively retrieve and utilize useful knowledge. Consequently, the search space may include noisy, low-quality, or overly domain-specific knowledge, and may also inherit biases from LLM corpora and research trends. For example, if the knowledge pool is largely constructed from papers published during a period dominated by Transformer-based improvements, the resulting search process may overemphasize that direction and narrow the scope of exploration. Figure 1: Overview of the proposed NAS. ( Top ) The model design attribute tree is generated from state-of-the-art (SOTA) models and a structured template. This tree is used to extract high-quality, fine-grained model design knowledge. ( Bottom ) FairNAD, a LLM-driven framework, then searches for high-performing models using mutation with fair idea sampling, Pareto-aware mutation, LLM-driven iterative mutation, and feedback loop. In this work, we address the inefficiency of open-ended NAS by proposing to structure the design knowledge for open-ended NAS. In contrast to prior open-ended methods such as [ 54 ] , which draws design knowledge from an unstructured knowledge pool, we curate the knowledge using a structured model design attribute template. This structured representation enables filtering of overly domain-specific knowledge, collection of higher-quality design ideas into a model design knowledge database, and mitigation of biases inherited from LLM priors and research trends. Further, in a vast open-ended search space, unstructured and performance-driven search strategies remain inefficient. In particular, strategies that sample design ideas from a flat knowledge pool, select parent architectures mainly according to validation performance, and discard candidates that violate HW constraints may suffer from three limitations: (1) they overlook diverse model candidates, particularly failing to select compact yet high-performing models for their size; (2) they lack intensive exploration of promising models; and (3) diversity is reduced by discarding models that exceed hardware (HW) constraints. Thus, building on the structured knowledge space, we propose FairNAD, a framework which employs a multi-stage search process for efficient and effective open-ended neural architecture design (NAD). First, FairNAD introduces the mutation with fair idea sampling stage which enables an unbiased and uniform exploration of a vast design space by retrieving design ideas based on the structured model knowledge pool. In particular, the idea is sampled according to its attributes, referred to as fair idea sampling. Second, the Pareto-aware mutation stage performs Pareto-aware sampling under HW constraints, together with upscaling and hyperparameter adjustment, so that compact yet promising models are not overlooked during the evolution. Third, the LLM-driven iterative mutation stage intensively refines the most promising models to accelerate convergence. Furthermore, the feedback loop is incorporated to downsize any models that exceed HW constraints, thereby encouraging the diversity of candidate models by retaining models that would otherwise be discarded. We demonstrate the effectiveness of FairNAD in designing high-performing architectures by exploring the open-ended search space efficiently and effectively. The experiments on NAS-Bench-201 achieves state-of-the-art performance over previous methods. Our contributions are summarized as follows: • We introduce a semi-automated model design attribute structuring method that organizes design knowledge into a hierarchical attribute tree. Based on this structured representation, we curate and organize diverse model design ideas from papers. • We introduce FairNAD, using a multi-type mutation designed to efficiently explore an open-ended search space. First, the mutation with fair idea sampling stage uniformly explores a wide range of model design ideas based on their design attribute. Second, the Pareto-aware mutation stage scales up compact yet high-performing models. Third, the LLM-driven iterative mutation stage intensively refines the most promising models. This multi-stage approach enhances both the diversity of candidate models and search efficiency. Furthermore, each model generation process incorporates a feedback loop . This loop verifies and refines generated models. In particular, models exceeding the HW constraints are scaled down which further promotes architectural diversity. 2 Related Works 2.1 Neural Architecture Search (NAS) Neural Architecture Search (NAS) aims to automatically explore optimal neural network architectures. Early NAS work demonstrated that reinforcement-learning controllers can generate competitive architectures [