Half Search Space is All You Need

Pavel Rumiantsev, Mark Coates · 2025

Read Half Search Space is All You Need by Pavel Rumiantsev, Mark Coates (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13586.txt.

Introduction

Pavel Rumiantsev and Mark Coates present a way to shrink the search for good neural network designs, in a 2025 paper. Automated design search is powerful, but one-shot methods, which train a shared super-network once, need huge graphics memory during the search. That cost limits who can use them.

Their fix prunes the search space before the one-shot search begins. A zero-shot method, which scores designs without training, removes weak candidates cheaply and automatically. The one-shot search then runs on the smaller space. On a standard design space, memory use fell by more than four fifths against the baseline setup, while final accuracy stayed the same. Search time also fell. The authors offer the two-stage approach for teams that want automated design without large hardware.

Catalog record

CREATORPavel Rumiantsev, Mark Coates
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Half Search Space is All You Need Pavel Rumiantsev, Mark Coates Abstract Neural Architecture Search (NAS) is a powerful tool for automating architecture design. One-Shot NAS techniques, such as DARTS, have gained substantial popularity due to their combination of search efficiency with simplicity of implementation. By design, One-Shot methods have high GPU memory requirements during the search. To mitigate this issue, we propose to prune the search space in an efficient automatic manner to reduce memory consumption and search time while preserving the search accuracy. Specifically, we utilise Zero-Shot NAS to efficiently remove low-performing architectures from the search space before applying One-Shot NAS to the pruned search space. Experimental results on the DARTS search space show that our approach reduces memory consumption by 81% compared to the baseline One-Shot setup while achieving the same level of accuracy. 1 Introduction Neural Architecture Search (NAS) is an approach to automate the design of a neural network. Unlike in a manual design, a researcher presents a set of all possible designs to the NAS algorithm, which searches for the best design. It is common for NAS not only to save time and computation usage, but also to outperform manual designs. The early approaches were characterised by a large computational overhead ( Elsken et al., 2019 ; Zoph et al., 2018 ) . Addressing this, Liu et al., 2018 introduced a One-Shot NAS paradigm (identifying the optimal architecture by a single training) by treating operation selection as a determination of the weights over a fixed operation set. All candidate operations (e.g., 3x3 convolution or maximum pooling) from a predefined set are evaluated as parallel branches. The weighted sum is taken over the outputs of all candidate operations. By making weights of the sum differentiable, the selection is represented as a differentiable operation. This enables the operation selection task to be addressed as an optimisation problem within an end-to-end differentiable framework, utilising efficient gradient backpropagation, resulting in improved performance with significantly reduced search costs ( Kang et al., 2023 ) . Although this design leads to a fast search with excellent performance, it suffers from excessive memory usage during the search stage as the gradients associated with all candidate operations should be backpropagated at the same time ( Xu et al., 2019 ) . In particular, this makes applying One-Shot NAS to large search spaces very memory demanding. Thus, for practical application, in a setting where there are strict memory constraints, a search space has to be very carefully designed before One-Shot search is applied. The aim is to make it as small as possible via manual design ( Chen et al., 2019 ) . The alternative search paradigm is to evaluate each architecture without an expensive optimisation procedure. Zero-Shot approaches estimate the quality of each candidate architecture using ranking functions ( Abdelfattah et al., 2021 ; Mellor et al., 2021 ) . These architecture fitness estimation approaches are fast and memory efficient compared to One-Shot NAS methods. In particular, the Zero-Shot techniques scale much more readily to larger search spaces compared to One-Shot NAS methods. Zero-Shot NAS approaches are generally good at identifying architectures with poor performance, but struggle to determine which of two top-performing architectures is better ( Mok et al., 2022 ; Ning et al., 2021 ) . Although some recent Zero-Shot approaches come close to competing with One-Shot NAS methodologies in performance ( Cavagnero et al., 2023a ; Li et al., 2022 ; Sun et al., 2023 ) , One-Shot NAS is still the more popular choice due to its greater reliability when applied to a new search space. Although Zero-Shot and One-Shot NAS complement each others’ strengths and weaknesses, they cannot be used simultaneously. However, a Zero-Shot method may be applied to significantly reduce the initial search space, pruning the architectures that are clearly poor candidates. After pruning, One-Shot NAS can be applied to the resulting reduced search space. The reduction of the search space reduces the memory requirements and speeds up the computation ( Wang et al., 2023 ) . Therefore, this combination requires fewer resources compared to a One-Shot method applied to the initial search space and produces more accurate results than using only Zero-Shot on the initial search space. The pipeline presented in this paper utilises both Zero- and One-Shot NAS. It is faster and more memory efficient than One-Shot NAS, while being more precise than the original Zero-Shot NAS. Contributions • We present a general Zero-One-Shot NAS framework that is designed to easily extend the applicability of One-Shot techniques to larger search spaces. • We demonstrate through experiments how our approach, when compared to One-Shot methods, significantly reduces memory and time consumption during the search phase while preserving the evaluation accuracy. 2 Related work 2.1 One-Shot architecture search Liu et al., 2018 proposed DARTS, the foundational idea for all One-Shot NAS techniques, by formulating the architecture search as an optimisation problem. It introduces a weighted sum (or mixing operation) across the candidate operation set, effectively transforming the architecture search to the task of identifying the optimal weight for each operation. The specialised search model, which is composed of multiple mixing operations, is commonly referred to as a supernetwork. The optimal network is derived by replacing each mixing operation with the operation that has the highest weight within its respective set. Chu et al., 2020 pointed out that the original design of DARTS suffers from stability and generalisation issues. Typically, these deficiencies are addressed by adding regularisation ( Chu et al., 2020 ; Zela et al., 2020 ; Chu et al., 2021 ; Ye et al., 2022 ; Movahedi et al., 2023 ) . Such an approach increases robustness while requiring minimal change to the existing framework. The core generalisation issue is addressed by Zela et al., 2020 via L_{2} regularisation on the parameters. Ye et al., 2022 address the stability issue by introducing beta-decay regularisation in \beta -DARTS. One of the recent works, \Lambda -DARTS ( Movahedi et al., 2023 ) tries to solve the generalisation issue by aligning gradients from different operators. Although regularisation improves robustness, it does not address computational complexity. Moreover, in some cases (e.g., \Lambda -DARTS), heavy regularisation increases complexity. Several works directly address the computational complexity and memory requirements of DARTS. Dong and Yang, 2019 present the idea of reducing computational complexity by estimating the operator distribution through sampling. PC-DARTS ( Xu et al., 2019 ) improves memory efficiency by selectively processing only a randomly selected portion of the input channels of a mixing operation (subsets of the network’s internal tensor representation). Cai et al., 2023 introduce EPC-DARTS, an updated version of PC-DARTS that incorporates an attention module to evaluate channel importance, allowing the selection of an important subset of channels instead of random ones to enhance performance. In a similar effort to optimise efficiency, FP-DARTS ( Wang et al., 2023 ) decomposes the operator search space into two separate disconnected search spaces, each explored within individual supernetworks. This separation reduces the computational burden required for backpropagation. Further improvement could be achieved by combining it with the idea of partial connection from PC-DARTS to minimise memory demands. Furthermore, Hu et al., 2024 propose a stability compensation module for PC-DARTS, tailored for

Read the complete text