From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Genetic Programming PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE
Quan Minh Phan, Ngoc Hoang Luong · 2025
Read From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Genetic Programming PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE by Quan Minh Phan, Ngoc Hoang Luong (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15832.txt.
Introduction
Cheap scores that guess a neural network's skill without training are usually hand-made, and a 2025 paper by Quan Minh Phan and Ngoc Hoang Luong evolves them instead. Hand design is slow and the scores wobble across tasks. Their frame uses genetic programming to breed formulas from parts of existing scores.
The evolved score tracks true network rankings well across many search spaces and tasks. Tests span thirteen problems from a public benchmark suite. The frame is built to extend, so new parts can join the breeding pool. Architecture searchers gain a faster, steadier filter for which designs deserve full training. That can cut the huge computing cost of hunting for good networks.
Catalog record
| CREATOR | Quan Minh Phan, Ngoc Hoang Luong |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15832
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Genetic Programming PubID: pubid: 0000–0000/00$00.00 © 2021 IEEE Quan Minh Phan, Ngoc Hoang Luong Abstract Estimating the network performance using zero-cost (ZC) metrics has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a notable limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Furthermore, the design of existing ZC metrics is manual, involving a time-consuming trial-and-error process that requires substantial domain expertise. These challenges raise two critical questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a more generalizable one? In this study, we propose a framework based on Symbolic Regression via Genetic Programming to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to true network performance across diverse NAS search spaces and tasks. Extensive experiments on 13 problems from NAS-Bench-Suite-Zero demonstrate that our automatically generated proxies consistently outperform hand-crafted alternatives. Using our evolved proxy metric as the search objective in an evolutionary algorithm, we could identify network architectures with competitive performance within 15 minutes using a single consumer GPU. {quanphm, hoangln}@uit.edu.vn I Introduction Fig. 1 : Comparisons in Kendall’s Tau rank correlations between our automatically-designed ZC metric and state-of-the-art ZC metrics (i.e., MeCo, ZiCo, and SWAP) on NAS-Bench-101/201/301 (Left) and TransNAS-Bench-101-Micro/Macro (Right). Highest Kendall’s \tau scores for each problem are presented. The results highlight the consistency of our ZC metric across various search spaces and tasks. Neural architecture search (NAS), a subfield of Automated Machine Learning (AutoML), focuses on the automatic design of high-performance network architectures [ 1 ] . NAS can be described as a procedure in which a search algorithm explores a search space containing potential architectures. During the exploration, the search algorithm assesses the quality of candidate networks using a performance estimation strategy . Early NAS studies estimated the quality of networks based on their performance on the validation dataset [ 2 , 3 ] . Although top-performing networks could be obtained, a single NAS run took several weeks or months to complete and heavily depended on hardware resource (i.e., 800 GPUs for a single run in [ 3 ] ). Recent works have proposed various approaches to deal with this bottleneck such as weight-sharing [ 4 ] , performance prediction [ 5 ] , or zero-cost metrics [ 6 ] . Among these techniques, using zero-cost metrics (also known as training-free metrics in the literature) offers the most efficiency since it can shorten the search time from days to a dozen seconds or minutes. For example, [ 7 ] found a network architecture with competitive performance on CIFAR-10 within 6 minutes using SWAP, which is a zero-cost proxy that measures the expressivity of the network. However, a significant limitation of the zero-cost metrics (ZC metrics) is their order-preserving ability . The order-preserving ability of ZC metrics can be assessed by examining the consistency between the rankings of architectures produced by ZC proxies and those based on their true performance [ 8 ] . Many studies have highlighted inconsistencies in network rankings based on ZC metrics’ scores across different problems [ 6 , 9 ] . While ZC metrics exhibit positive rank correlations with network performance on some problems, they may demonstrate negative correlations on others. Recent studies have attempted to address this issue when newly proposed ZC proxies (e.g., ZiCo [ 10 ] , MeCo [ 11 ] , SWAP [ 7 ] ) achieve positive rank correlations across all testing problems. In these cases, a network that receives a high ZC score is also likely to exhibit high performance, and vice versa . However, despite achieving positive rank correlations, the scores for some problems remain close to 0.0, indicating a lack of correlation between the ZC scores and the networks’ performance. In this study, our target is to design a ZC metric that yields not only the consistent order-preserving ability but also a high positive rank correlation across various problems 1 1 1 In this paper, we define an NAS problem consisting of a single search space (e.g., NAS-Bench-101, DARTS) and a single task (e.g., image classification on CIFAR-10 dataset, image classification on ImageNet dataset, object detection on Taskonomy dataset). If we search network architectures within one search space but evaluate them on two different tasks, we consider that we are solving two NAS problems. We list all 13 NAS problems in our experiments in Appendix D of the Supplementary. . TABLE I : List of frameworks that automatically design ZC metrics. Name Method Features Multiple- problems High- extensibility EZNAS [ 12 ] Symbolic Regression Low-level ✗ ✗ Auto-Prox [ 13 ] Symbolic Regression Low-level ✔ ✗ AZNAS [ 14 ] - High-level ✗ ✔ UP-NAS [ 15 ] Neural network High-level ✗ ✔ RoBoT [ 16 ] Bayesian Optimization High-level ✗ ✔ Ours Symbolic Regression High-level ✔ ✔ Most proposed ZC metrics are manually designed with expert knowledge. Can we automate the design of efficient ZC metrics to form the proxies that are better than hand-crafted ones? The practicability of this approach has been demonstrated in several studies. EZNAS [ 12 ] implemented Symbolic Regression to form a ZC metric by combining three low-level features of networks: weights, activations, and gradients. While EZNAS only considered convolutional neural networks, Auto-Prox [ 13 ] extended the framework to design ZC metrics for vision transformer architectures. Additionally, a single run of Auto-Prox aims to search for a ZC metric that could perform well across multiple problems instead of a single problem as in EZNAS. UP-NAS [ 15 ] designed a new ZC metric by combining higher-level features of networks (i.e., hand-crafted ZC metrics) and taking their weighted sum, in which the weights of metrics are found by using a neural network. However, a search process is needed to obtain suitable weights whenever we handle a new problem. In this paper, we focus on the automatic designing of efficient ZC metrics. Table I highlights the differences between our proposed framework and previous approaches that also automate the design of ZC metrics. We discuss in detail the differences in Section III . Generally, our framework is designed to synthesize a new ZC metric from existing hand-crafted ones, and the outcome metric could perform well across diverse problems. Our method offers high extensibility, making it straightforward to adapt to various problems or modify input features to enhance the performance of the resulting metric. Our main contributions are as follows: • We propose a highly extensible framework for automatically designing a ZC metric that not only consistently preserves the order of network performance but also achieves a high positive rank correlation across multiple problems. • Our framework can synthesize a new ZC metric within only 10 minutes, which is significantly faster than the 24 hours required by EZNAS. Compared to previous ZC metrics, including newly proposed ones (i.e., ZiCo, MeCo, SWAP), our metric achieves the state-of-the-art Kendall’s tau correlation for NAS-Bench-101/201/301 search spaces and comparable scores for TransNAS-Bench-101-Micro/Macro search