Meta knowledge assisted Evolutionary Neural Architecture Search

Yangyang Li, Guanlong Liu, Ronghua Shang, and Licheng Jiao · 2025

Read Meta knowledge assisted Evolutionary Neural Architecture Search by Yangyang Li, Guanlong Liu, Ronghua Shang, and Licheng Jiao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21545.txt.

Introduction

Yangyang Li, Guanlong Liu, Ronghua Shang, and Licheng Jiao offer a 2025 method for designing neural networks by computer search. Search methods based on evolution can find strong designs, but testing each design costs a great deal of time. A fixed training pace also loses more detail on some designs than on others.

Their fix uses prior knowledge to set a better training pace for each design. A stand-in model picks the most hopeful designs early, then tests them in full. A change step that repeats on a cycle adds more variety to the pool. Tests on three image sets matched top methods at lower cost.

Catalog record

CREATORYangyang Li, Guanlong Liu, Ronghua Shang, and Licheng Jiao
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-21545

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Meta knowledge assisted Evolutionary Neural Architecture Search Yangyang Li, Guanlong Liu, Ronghua Shang, and Licheng Jiao Abstract Evolutionary computation (EC)-based neural architecture search (NAS) has achieved remarkable performance in the automatic design of neural architectures. However, the high computational cost associated with evaluating searched architectures poses a challenge for these methods, and a fixed form of learning rate (LR) schedule means greater information loss on diverse searched architectures. This paper introduces an efficient EC-based NAS method to solve these problems via an innovative meta-learning framework. Specifically, a meta-learning-rate (Meta-LR) scheme is used through pretraining to obtain a suitable LR schedule, which guides the training process with lower information loss when evaluating each individual. An adaptive surrogate model is designed through an adaptive threshold to select the potential architectures in a few epochs and then evaluate the potential architectures with complete epochs. Additionally, a periodic mutation operator is proposed to increase the diversity of the population, which enhances the generalizability and robustness. Experiments on CIFAR-10, CIFAR-100, and ImageNet1K datasets demonstrate that the proposed method achieves high performance comparable to that of many state-of-the-art peer methods, with lower computational cost and greater robustness. PubID: pubid: Copyright ©2025 IEEE. Personal use of this material is permitted. However, permission to use this material for any other purposes must be obtained from the IEEE by sending an email to [email protected]. I Introduction The accelerated development of deep learning has achieved remarkable success in many challenging real-world tasks, such as image classification [ 1 ] , object tracking [ 2 , 3 ] and natural language processing [ 4 ] . Neural networks, such as ResNet [ 5 ] and MobileNetV2 [ 6 ] , are always derived from considerable manual experience. Nevertheless, some neural networks are only applicable to a single dataset, and the manually crafted architectures often present challenges for common users, as they require the expertise of human specialists in both the deep neural networks (DNNs) and the relevant problem domain to design effectively. Recently, automated machine learning (AutoML) has gained increasing attention from both industry and academia [ 7 , 8 , 9 ] . Neural architecture search (NAS), a cutting-edge technology in AutoML, aims to automate the design of DNNs with better performance, especially under complex constraints in real-world tasks. In general, there are numerous methods that can be used for NAS. For example, reinforcement learning (RL)-based methods [ 10 , 11 , 12 ] , gradient descent (GD)-based methods [ 13 , 14 , 15 ] , and evolutionary computation (EC)-based methods [ 16 , 17 , 18 ] are popular in architecture search. RL-based algorithms construct the model via a reward function, which is designed on the basis of an evaluation of the model’s performance. While this type of method can achieve high accuracy, it is often extremely computationally expensive. For example, on the widely used small natural image dataset CIFAR-10, NAS-RL [ 10 ] requires simultaneously running for thousands of GPU days, which poses an insurmountable challenge for the majority of researchers and devices. GD-based algorithms consider architecture search to be a continuous optimization problem [ 19 ] , which can search more efficiently because of the continuously differentiable operation space. However, algorithms such as DARTS [ 13 ] always exhibit unstable performance, and their search space is restricted by the predefined operation space. EC-based algorithms have received extensive attention because of their ability to solve nondifferentiable problems as well as their global search ability to handle complex constraints and optimization problems. Real et al. [ 20 ] introduced a large-scale evolution framework to explore novel architectures spanning 2750 GPU days. In [ 21 ] , Sun et al. proposed a method named CNN-GA for the automatic design of convolutional neural network (CNN) architectures using genetic algorithms. Although it took up to 35 GPU days to find the best architecture, it helped people without domain knowledge of the CNN obtain promising CNN architectures for a given image. Sun et al. [ 22 ] proposed AE-CNN, a CNN architecture that evolved by incorporating two valid blocks. It took approximately 27 GPU days to obtain an optimized CNN structure on CIFAR-10 [ 23 ] . NSGANet, which explores the space of feasible architectures through a three-step process and can be transferred to multiple scenarios, was proposed by Lu et al. [ 24 ] . As a result, it discovered an architecture that achieved an accuracy rate of 97.25% using 4 GPU days. Zhou et al. [ 25 ] proposed a knowledge-sharing mechanism in multitask scenarios and achieved competitive performance with a 2 \times lower search cost. Zhao et al. [ 26 ] presented a knowledge-guided adaptive multiobjective evolutionary strategy for remote sensing classification that effectively combines the strengths of CNN and Transformer via the knowledge learned from natural images. [ 27 ] proposed a concept of supernet shifting to improve order-preserving ability, but it focuses primarily on fine-tuning the supernet. To address the major computational cost limitation, significant efforts have been made to increase computational efficiency using multiple techniques, including weight inheritance [ 28 , 29 ] , weight sharing [ 30 , 31 ] , surrogate model [ 32 , 33 , 34 ] , and population reduction [ 16 ] . An appropriate learning rate (LR) schedule can significantly improve a model’s generalizability and DNN training results. Wu et al. [ 35 ] also theoretically highlighted the unique role of the LR schedule in global minimum selection. Meta-learning enables models to automatically acquire appropriate learning rates by optimizing the adaptation process across multiple tasks. Instead of manually tuning fixed learning rates, the meta-learning framework meta-SGD [ 36 ] treats the learning rate itself as a learnable parameter within a bilevel optimization process. The meta-learner optimizes the learning rate across tasks by minimizing a meta-objective. The model then performs task-specific updates using gradient descent with a learning rate guided by the meta-learner. While the meta-learning mechanism can make the model more suitable for the data and significantly improve the model’s generalizability and robustness, traditional meta-learning strategies are usually computationally expensive and focus on optimizing specific neural architectures, which makes applying meta-learning to NAS difficult. To solve the abovementioned problems, we design a novel and efficient EC-based NAS method through a meta-learning framework, termed MetaNAS. The proposed method achieves a good balance between computational cost and model performance by designing an efficient learning rate (LR) schedule in advance during pretraining, utilizing a meta-learning rate (Meta-LR) scheme. Each individual is then evaluated using the tailored LR schedule. Unlike traditional meta-learning algorithms, which always take the gradients as input, the Meta-LR scheme learns only the LR schedule, which avoids the shortcomings of traditional meta-learning algorithms, which are computationally intensive and difficult to generalize to different architectures. The Meta-LR scheme can be easily used in existing EC-based NAS methods to increase their robustness and reduce complexity. We introduce an adaptive surrogate model with an adaptive threshold, which can predict potentially superior architectures in a few epochs and evaluate the selected architectures with complete epochs. In addition, we design a period mutation operator in the evolutionary algorithm to increase population diversity and improve the generalizability, making it more in line with the current trend

Read the complete text