Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search CCS: Computing methodologies Genetic algorithms CCS: Computing methodologies Neural networks CCS: Computing methodologies Supervised learning by regression CCS: Computing methodologies Search methodologies
Pranav Somu, Advay Balakrishnan, Stepan Kravtsov, Aaron McDaniel, Jason Zutty · 2026
Read Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search CCS: Computing methodologies Genetic algorithms CCS: Computing methodologies Neural networks CCS: Computing methodologies Supervised learning by regression CCS: Computing methodologies Search methodologies by Pranav Somu, Advay Balakrishnan, Stepan Kravtsov, Aaron McDaniel, Jason Zutty (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15649.txt.
Introduction
Pranav Somu, Advay Balakrishnan, Stepan Kravtsov, Aaron McDaniel, and Jason Zutty use frozen language models as trait extractors for neural architecture search. Architectures are fed in as PyTorch class text. No search-specific tuning is needed. The traits pass through a light regression head to form the predictor. Raw code inputs beat other text encodings on two search spaces. On one space, swapping path encodings for code traits cuts the test budget to land within one percent of the best architecture by 34 percent for CIFAR-100 scores.
Catalog record
| CREATOR | Pranav Somu, Advay Balakrishnan, Stepan Kravtsov, Aaron McDaniel, Jason Zutty |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52503 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.15649
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Towards Code-Oriented LM Embeddings for Surrogate-Assisted Neural Architecture Search CCS: Computing methodologies Genetic algorithms CCS: Computing methodologies Neural networks CCS: Computing methodologies Supervised learning by regression CCS: Computing methodologies Search methodologies Pranav Somu, Advay Balakrishnan, Stepan Kravtsov, Aaron McDaniel, Jason Zutty Abstract . Developing effective surrogates (performance predictors) for Neural Architecture Search (NAS) typically requires expensive fine-tuning or the engineering of complex representations. We propose a low-cost embedding strategy that leverages the inductive bias of Language Models (LMs) to eliminate these overheads. By representing architectures as PyTorch class definition text, we demonstrate that off-the-shelf LMs act as competitive feature extractors without NAS-specialized fine-tuning. The final predictor is constructed by passing the extracted Code-Oriented LM Embeddings (COLE) through a lightweight regression head. We also investigate strategies to improve embedding quality and utilization. Our experiments on the NAS-Bench-201 and einspace search spaces reveal that raw code inputs yield higher predictive performance than other text-based encodings (e.g., ONNX-to-text encodings) when using frozen LMs. We also observe COLE drives superior surrogate-assisted search using the BANANAS algorithm in NAS-Bench-201. When optimizing for CIFAR-100 performance, replacing structural path encodings with COLE for architecture representation allows for a 34% decrease in the evaluation budget required to reach within 1% of the fittest architecture in the search space (by test accuracy). As any neural architecture can be represented as code, these findings establish COLE as a versatile and efficient foundation for advancing NAS. 1 1 1 This is an extended version of work accepted to GECCO 2026: https://doi.org/10.1145/3795101.3805435 2 2 2 In the official print, we reported the CIFAR-100 evaluation metric as ‘validation accuracy,’ but we correct the terminology to ‘test accuracy’ in this print to accurately reflect the evaluation data provided by the NAS library we used. 3 3 3 Our code is available at https://github.com/pcsom/cole/tree/v1.0 1. Introduction Neural Architecture Search (NAS) automates the discovery of high-performing neural architectures for a given task in a search space. However, conducting NAS in large search spaces becomes computationally expensive due to complex evaluation functions. While the objective for identifying an optimal architecture in a search space is simple to formulate (e.g., minimizing a loss metric), the cost of evaluating the fitness function renders brute force searches computationally intractable. This computational bottleneck motivates the area of performance predictors (surrogates), which seek to approximate the fitness function using low-cost estimators ranging from simple Learning Curve Extrapolation ( 24 ) to complex Graph Neural Networks (GNNs) ( 2 ) , effectively substituting the expensive ground-truth evaluations with quick inferences. For this method, architectures must be converted into input representations compatible with surrogate models. Structural representations such as adjacency matrices or path encodings capture network topology sufficiently. However, capturing hyperparameters like convolution kernel sizes requires custom strategies, such as registering all acceptable values as separate configurations, which is often complex to implement or computationally inefficient ( 14 ) . Furthermore, these encoding schemes are inherently rigid, typically relying on fixed-dimensional matrices tailored to specific cell-based search spaces. This means applying the predictor to novel architecture spaces requires re-engineering of the encoding scheme to accommodate varying properties like quantity and arrangements of operations. Recent efforts to make generalized encodings have turned to Language Models (LMs), proposing them as universal regressors. Notable frameworks such as in Qin et al. ( 15 ) have attempted to unify NAS by converting architectures into text-based encodings and fine-tuning encoder-based language models (e.g., ModernBERT) to interpret them. Qin et al. ( 15 ) translated einspace ( 4 ) , an expressive search space which builds architectures from fundamental operations, into derivation tree strings. These strings were used to fine-tune ModernBERT Large end-to-end to predict architecture performance. While effective, the method introduces the high computational cost of fine-tuning the LM to learn a foreign representation format. By requiring a warm-up phase of fine-tuning, such methods counteract the efficiency gains of using a surrogate model. Instead, it would be beneficial for a search algorithm to identify promising regions of the search space out-of-the-box, enabling cold-start searches. A key insight that drives our work is that neural architectures naturally have a code representation. Language models like CodeLlama and ModernBERT have been pre-trained on trillions of tokens, some (or most) of which are code ( 5 ; 19 ) . In contrast, text-based encodings like derivation tree strings or ONNX-to-text encoding ( 14 ) are underrepresented in training data thus misaligned with the models’ learned distributions. We demonstrate that native code representations utilize the LM’s out-of-the-box interpretation capabilities more effectively than other misaligned text-based formats. In this work, we investigate the efficacy of using frozen LMs as zero-shot embedding engines for NAS. Unlike prior approaches that rely on expensive fine-tuning and text-based encodings, we feed raw PyTorch class definition text directly into frozen LMs and use the model’s hidden state activations to compute a Code-Oriented LM Embedding (COLE). Code can represent any neural architecture, so our work can be incorporated into most surrogate-enabled NAS pipelines. Our contributions are as follows: (1) We show that COLE is an embedding strategy superior to other text-based encodings with frozen LMs. We demonstrate that computationally expensive fine-tuning such as in Qin et al. ( 15 ) may not be strictly necessary to achieve effective NAS embeddings. (2) We demonstrate that integrating COLE as a neural architecture representation strategy into an existing surrogate-assisted NAS (SA-NAS) algorithm leads to improved performance compared to representing architectures using structural encodings like path encodings. (3) We document effective tuning applicable to the COLE and regression pipeline, including PyTorch code verbosity and Principal Component Analysis (PCA) for dimensionality reduction. By validating that frozen LMs, coupled with code as input, can serve as neural architecture embedders out of the box, we pave the way for a search paradigm that is not only search-space agnostic but also removes the expensive pre-training that is required to learn specialized formats of architectures. 2. Related Work 2.1. Search Spaces and NAS Benchmarks Search spaces are commonly defined by a cell-based approach, where the cell is a repeatable unit (a block of operations) stacked in groups to form a large architecture ( 28 ; 26 ; 3 ; 27 ; 8 ) . In a cell-based approach, the search algorithm optimizes the cell rather than the entire network topology. Each cell is a directed acyclic graph (DAG) with a predefined operation set between nodes. For example, cells of the NAS-Bench-201 search space ( 3 ) contain zeroize, skip-connect, convolution, and average pooling operations. More recently, there has been work on defining search spaces based on context-free grammars ( 4 ; 16 ) . Einspace, for example, leverages a functional grammar to use fundamental operations to construct architectures, represented as derivation trees. This allows discovery of novel primitives that flexibly derive and combine elements of CNNs, Transformers and MLPs. To facilitate reproducible comparison of surrogate models or NAS algorithms, there are widely adopted NAS benchmarks ( 26 ; 3 ; 27 ) . These benchmarks provide pre-computed architecture-accuracy pairs within their own defined search spaces, allowing easy evaluation during a NAS algorithm execution. We conduct our evaluations using NASLib ( 21 ) , a modular open-source framework that integrates these benchmarks and provides a unified interface for search strategies. Specifically,