LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding
Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng Li · 2025
Read LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding by Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-05657.txt.
Introduction
Language models are taught to design neural networks across task domains by Yuxuan Hu and nine co-authors in a 2025 machine learning paper. Architecture search is usually tuned per domain with heavy prompt craft, limiting how far language model approaches scale.
Their framework encodes any architecture as a universal string of numbers, letting one representation travel across domains. The search problem becomes ranking: the model learns to pick strong candidates from a pool, trained on examples built by a pruning-based sampling of the design space. Experiments cover image classification networks and adapter settings for segmentation and generation. Performance stayed competitive in and out of domain.
Catalog record
| CREATOR | Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-05657
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding Yuxuan Hu, Jihao Liu, Ke Wang, Jinliang Zheng, Weikang Shi, Manyuan Zhang, Qi Dou, Rui Liu, Aojun Zhou, Hongsheng Li Abstract Recent progress in Large Language Models (LLMs) has opened new avenues for solving complex optimization problems, including Neural Architecture Search (NAS). However, existing LLM-driven NAS approaches rely heavily on prompt engineering and domain-specific tuning, limiting their practicality and scalability across diverse tasks. In this work, we propose LM-Searcher, a novel framework that leverages LLMs for cross-domain neural architecture optimization without the need for extensive domain-specific adaptation. Central to our approach is NCode, a universal numerical string representation for neural architectures, which enables cross-domain architecture encoding and search. We also reformulate the NAS problem as a ranking task, training LLMs to select high-performing architectures from candidate pools using instruction-tuning samples derived from a novel pruning-based subspace sampling strategy. Our curated dataset, encompassing a wide range of architecture-performance pairs, encourages robust and transferable learning. Comprehensive experiments demonstrate that LM-Searcher achieves competitive performance in both in-domain (e.g., CNNs for image classification) and out-of-domain (e.g., LoRA configurations for segmentation and generation) tasks, establishing a new paradigm for flexible and generalizable LLM-based architecture search. The datasets and models will be released at https://github.com/Ashone3/LM-Searcher . 1 Introduction Recent advances in Large Language Models (LLMs) have demonstrated remarkable potential in solving complex optimization problems, including competitive programming OpenAI 2024 , the Traveling Salesman Problem (TSP) Yang et al. 2023 , and even Neural Architecture Search (NAS) Zheng et al. 2023 ; Nasir et al. 2024 . Among these applications, NAS focuses on discovering optimal neural architectures that maximize performance in a given domain. Previous approaches, such as GENIUS and LLMatic, leverage GPT-4 OpenAI 2023 to generate task-specific architectures, either in code or natural language form. These preliminary attempts focus on designing prompts to instruct off-the-shelf LLMs for the architectural design of image classification models. However, prompt engineering-based approaches require extensive domain-specific expertise and manual prompt tuning to handle different search spaces and tasks, making them less practical in real-world scenarios. For instance, while GPT-4o can effectively identify better architectures within a simple classification search space such as CIFAR-10, it fails to surpass the random search approach when applied to larger search spaces like Diffusion Transformer Peebles and Xie 2023 for image generation. In this paper, we ask the following question: Can LLMs be trained as general-purpose search models capable of optimizing neural architectures across diverse domains, without the need for domain-specific tuning? To answer this question, we introduce LM-Searcher, a novel framework that leverages LLM’s reasoning and optimization capabilities for exploring neural architecture, which transforms neural architectures into task-agnostic universal representations, enabling cross-domain search using either off-the-shelf LLMs or a fine-tuned, search-specific LLM. Specifically, architectures are first represented as numerical strings termed NCode for simplicity, and inputed into the LLM for architecture encoding. As illustrated in Fig 1 , architecture configurations are represented by their sub-module indices in the search space. This encoding method unifies tasks across different domains by representing them as combinatorial optimization problems. For example, in NAS-Bench-201 Dong and Yang 2020 , each architecture is built from "cell" units with five configurable operations, encoded as: ["0" (zeroize), "1" (skip connection), "2" (1×1 conv)…]. Unique architectures are represented by combinations of these operation codes. Such numerical encoding of architectures not only bridges the gap between different architecture domains, but also directs the model’s attention towards analyzing inter-architecture relations based on their performance, minimizing distractions from domain-specific characteristics. Subsequently, we extract architecture-performance metadata from established NAS benchmarks Ying et al. 2019 ; Zela et al. 2020 , which contains up to 10^{18} possible NCode-accuracy pairs (e.g., [NCode: 333123, Accuracy: 91.45%]). A straightforward approach is to directly use the accuracy of known historical NCode-accuracy pairs to predict better-performing architectures. However, due to the vastness of the neural architecture search space and the large-scale possible combinations of historical and predicted architectures, it is challenging for LLMs to generate the optimal architecture directly. To mitigate this challenge, we reformulate the task as a ranking problem rather than a generation problem, i.e., the LLM is trained to always choose the next best architecture from a candidate pool. To effectively construct instruction-tuning samples that conform to our proposed encoding scheme and enhance the LLM’s capabilities to explore neural architectures, we introduce a novel pruning-based data curation strategy. This strategy prunes a fully-connected supernet to obtain various sub-networks, following the approach of Pham et al. 2018 . Prior works on pruning Liu et al. 2018b and weight-sharing NAS Xu et al. 2019 have demonstrated that architectures within a reduced sub-network can converge faster and achieve performance comparable to, or even better than, their larger counterparts. Based on this, we sample 100–200 architectures from the same subspace for both the history (NCode-performance) and candidate (NCode) sets, and select the highest-performing architecture as the ground truth. In total, we obtain a 228k instruction-tuning dataset for LLM training. Training on this carefully curated dataset allows our LLM to learn robust and transferable architecture-performance patterns. As a result, our LM-Searcher can effectively search for architectures not only in image classification tasks but also in previously unseen search spaces. Through comprehensive evaluation of LM-Searcher across diverse tasks without specific tuning, we demonstrate the effectiveness of the proposed LM-Searcher model in in-domain CNN architecture search for image classification (e.g. MobileNetV2) and out-of-domain model design (e.g., LoRA configuration for visual segmentation and generation tasks), respectively. Our main contributions can be summarized as follows: • To the best of our knowledge, we introduce NCode , the first cross-domain numerical string representation for neural architectures, enabling architecture optimization across diverse domains. • We develop LM-Searcher, a task-agnostic architecture search model trained using a novel pruning-based subspace sampling data curation mechanism. • We demonstrate the effectiveness of LM-Searcher across both in-domain CNN architecture search and out-of-domain model design tasks, highlighting its flexibility and generalizability. Figure 1: Architecture configurations are encoded as simplified numerical codes and provided to the LLM, which predicts the most promising candidate. This code is then decoded back into a neural architecture for performance evaluation. 2 Related Work 2.1 LLMs Solving Complex Problems Large language models (LLMs), pre-trained on massive datasets, have demonstrated remarkable proficiency in few-shot learning scenarios ( Brown et al. 2020 ) . The field has witnessed rapid advancements in LLM development, with notable contributions including models like LLaMA ( Touvron et al. 2023 ) , PaLM ( Chowdhery et al. 2023 ) , and GPT-4 ( OpenAI 2023 ) . Subsequent research has focused on fine-tuning these models, particularly open-source models like LLaMA, on specialized or synthesized datasets. This has led to significant performance improvements in areas such as code generation ( Roziere et al. 2023 ) and mathematical reasoning ( Wang et al. 2024 ; Gou et al. ) . Our work builds upon this trend by exploring the potential of LLMs for the complex task of neural architecture search (NAS). Our approach leverages a carefully curated dataset derived from existing NAS benchmark. This allows LM-Searcher to achieve strong cross domain generalization in NAS tasks, setting a new benchmark for LLM performance in this area. 2.2 Architecture Search with LLMs LLMs have