Neural Architecture Search by Learning a Hierarchical Search Space
Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli · 2025
Read Neural Architecture Search by Learning a Hierarchical Search Space by Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21061.txt.
Introduction
Searching the space of neural network designs is like exploring a game tree, and Monte Carlo tree search is a strong explorer when the order of choices is good. Three authors apply that search to designing image classification networks, studying how the sampling and branching choices affect what it finds.
Their key move is to teach the search its own map. They group candidate networks by how similar their outputs are and cluster them into a hierarchy. The tree then branches along that hierarchy, so early choices separate promising families from dead ends. On two demanding benchmarks, CIFAR-10 and ImageNet, the method found strong designs faster than rival approaches when the branching was learned this way. The authors present the result as evidence that in design search, learning how to organize the options matters as much as the search itself.
Catalog record
| CREATOR | Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21061
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Neural Architecture Search by Learning a Hierarchical Search Space Mehraveh Javan Roshtkhari, Matthew Toews, Marco Pedersoli Abstract Monte-Carlo Tree Search (MCTS) is a powerful tool for many non-differentiable search related problems such as adversarial games. However, the performance of such approach highly depends on the order of the nodes that are considered at each branching of the tree. If the first branches cannot distinguish between promising and deceiving configurations for the final task, the efficiency of the search is exponentially reduced. In Neural Architecture Search (NAS), as only the final architecture matters, the visiting order of the branching can be optimized to improve learning. In this paper, we study the application of MCTS to NAS for image classification. We analyze several sampling methods and branching alternatives for MCTS and propose to learn the branching by hierarchical clustering of architectures based on their similarity. The similarity is measured by the pairwise distance of output vectors of architectures. Extensive experiments on two challenging benchmarks on CIFAR10 and ImageNet show that MCTS, if provided with a good branching hierarchy, can yield promising solutions more efficiently than other approaches for NAS problems. 1 Introduction Neural Architecture Search (NAS) aims to automate neural architecture design and has shown great success in the recent years [ 1 , 2 , 3 , 4 ] , surpassing manually designed Convolutional Neural Networks (CNN) in deep learning [ 5 , 6 , 7 ] . NAS aims at yielding the best architecture within a given search space with a lower computational budget than a brute-force approach, based on training all possible architectures independently. One prominent solution is one-shot methods based on weight sharing [ 8 ] in which multiple architectures share all or part of their weights. These methods circumvent the need for individual training of each architecture by training a single "supernet" that contains all possible operation/architectural choices. Each architecture are evaluated by using weights inherited from that supernet. When architectures are compatible, weight-sharing allows to recycle training iterations [ 9 , 8 , 10 ] ; however, for vastly different architectures, it may lead to interference, i.e. weights beneficial to one harm others and vice-versa [ 11 ] . To reduce the detrimental effect of interference in weight sharing, prior work has explored either multiple specialized models that can focus on different parts of the search space [ 12 , 13 , 14 , 11 ] ; or importance sampling [ 5 , 15 , 16 ] in which the probability of an architecture performing well is estimated during training. In the latter, promising architectures are sampled more frequently during the course of training, gradually reducing the possible interference [ 5 , 17 ] as the training is guided towards the best architectures. Unlike uniform sampling [ 7 , 11 ] , the challenge of using importance sampling is to robustly estimate and identify superior architectures as early as possible in the training cycle. This would allow the model to focus on these architectures and to minimize wasting training resources on unpromising architectures. This requires fast and reliable estimation of the probability distribution of architectures in as few training iterations as possible. (independent) (joint) (conditional) Figure 1: Probability factorization of 8 architectures. We show different ways to approximate the discrete probability distribution of architectures for a toy example of search space with N=3 nodes (a,b,c in the figure) each one with O=2 possible operations for a total of 2^{3} architectures. (left) Assuming the nodes independent (as in DARTS [ 5 ] ) allows the model to estimate only N\times O probabilities. (center) Considering the joint probabilities would require to estimate O^{N} different probabilities (as in Boltzmann sampling). (right) The joint probability can be factorized into the product of conditional probabilities (in a hierarchy such as in MCTS). This does not reduce the probabilities to estimate, but allows a more efficient exploration of the search space. A neural network can be viewed as a graph, composed of nodes, which define the architecture, connected by edges. These nodes offer a choice of operations, which are the processes applied to the data (e.g. convolution, fully connected, etc.). In importance sampling, a common assumption for more efficient estimation of architecture probability is "node independence", i.e. considering nodes as statistically independent variables. For instance, in a neural network, the choice of the operation for the second layer would not depend on the choice of the operation in the first layer. This simplifies architecture probability estimation to a product of nodes’ probabilities (see Fig. 1 (independent)). This reduces the scale of the problem to learning individual node probabilities. While the widely used differentiable NAS method (DARTS [ 5 ] and followup works [ 16 , 18 , 15 ] ) rely on this assumption, overlooking the joint contribution of nodes to architecture performance can lead to a poor node selection for the final architecture [ 19 , 20 ] . In order to remove the node independence assumption, joint probabilities of all configurations should be estimated (see Fig. 1 (joint)). This requires estimating each architecture’s probability independently. In mainstream single-path one-shot (SPOS) methods [ 7 , 21 , 22 , 23 ] , at each iteration, one architecture from the supernet is sampled, trained and estimated and that estimation is used to update the probability distribution. With node independence, at each iteration several associated node probabilities are updated; while for joint probability only the probability of sampled architecture is be updated. Thus, the full update of the estimation costs proportionally to the number of architectures, making it unscalable to large search spaces. This inefficient estimation slows down the entire training as the importance sampling will not be able to focus on the promising architectures. To explore more wisely, a compelling option is to factorize the joint probability into conditionals, such as a tree structure for Monte-Carlo Tree Search (MCTS) (Fig. 1 (conditional)). While the number of probabilities to estimate remains the same as joint probability, it offers several key advantages. A well-designed hierarchical search space enables more efficient tree traversal by reducing the unnecessary exploration of unpromising branches. Prioritizing promising branches can lead to faster convergence, improved solution quality and better scalability. However, standard predefined hierarchies do not guarantee a more efficient exploration, as they are defined by construction and without taking into account the semantic similarity of the corresponding architectures. While in many MCTS problems the search hierarchy is defined by the sequentiality of the problem (e.g the moves in chess), for NAS there is no constraint in the order of exploring architectures. [ 24 ] and [ 25 ] leverage this by using the classification accuracy of the nodes for partitioning the search space into "good" and "bad" nodes to reduce the unnecessary exploration. This approach works well when running the search with a fixed, already learned, recognition model (i.e. the CNN weights). In fact, [ 24 ] uses MCTS for searching the best performing model on a supernet pre-trained with uniform sampling, while [ 25 ] perform MCTS for NAS using the already trained models provided by NAS-Bench-201 [ 26 ] . In this work, we tackle the more challenging problem of learning the recognition model and the tree