Sim-is-More: Randomizing HW-NAS with Synthetic Devices

Francesco Capuano, Gabriele Tiboni, Niccolò Cavagnero, Giuseppe Averta · 2025

Read Sim-is-More: Randomizing HW-NAS with Synthetic Devices by Francesco Capuano, Gabriele Tiboni, Niccolò Cavagnero, Giuseppe Averta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-00663.txt.

Introduction

Capuano and co-authors present a cheaper way to search for neural network designs in a paper from two thousand twenty-five. Hardware-aware search tailors a network to a device, but it usually needs a good model of that device's speed. Errors in that model can lead to poor choices.

Their two-stage method first trains a controller on many made-up devices. The controller then works directly on the real target device, using only a handful of true speed measurements. Training uses free accuracy proxies, so no full network training is needed in the first stage. Tests on a standard benchmark showed the method adapts to unseen devices and finds fast designs with very few probes.

Catalog record

CREATORFrancesco Capuano, Gabriele Tiboni, Niccolò Cavagnero, Giuseppe Averta
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-00663

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Sim-is-More: Randomizing HW-NAS with Synthetic Devices Francesco Capuano, Gabriele Tiboni, Niccolò Cavagnero, Giuseppe Averta Abstract Existing hardware-aware NAS (HW-NAS) methods typically assume access to precise information circa the target device, either via analytical approximations of the post-compilation latency model, or through learned latency predictors. Such approximate approaches risk introducing estimation errors that may prove detrimental in risk-sensitive applications. In this work, we propose a two-stage HW-NAS framework, in which we first learn an architecture controller on a distribution of synthetic devices, and then directly deploy the controller on a target device. At test-time, our network controller deploys directly to the target device without relying on any pre-collected information, and only exploits direct interactions. In particular, the pre-training phase on synthetic devices enables the controller to design an architecture for the target device by interacting with it through a small number of high-fidelity latency measurements. To guarantee accessibility of our method, we only train our controller with training-free accuracy proxies, allowing us to scale the meta-training phase without incurring the overhead of full network training. We benchmark on HW-NATS-Bench, demonstrating that our method generalizes to unseen devices and searches for latency-efficient architectures by in-context adaptation using only a few real-world latency evaluations at test-time. 1 Introduction Designing Deep Neural Networks (DNNs) for deployment on resource-constrained devices involves the dual challenge of (i) selecting architectures that perform well on a given task while (ii) respecting hardware constraints. Hardware-aware Neural Architecture Search (HW-NAS) addresses this challenge by automating architecture design under hardware supervision, offering an automated alternative to the otherwise-manual design of architectures that are efficient for different deployment scenarios. Still, most HW-NAS methods are tailored to a single target deployment scenario, requiring re-searching an architecture satisfying (i-ii) when the deployment scenario—in this work, the device d used to run DNNs—changes. In this work, we address this limitation focusing on multi-device HW-NAS. In this, our goal is to discover architectures performing well on (i) task-specific metrics—in this work, validation accuracy — (ii) hardware-efficiency metrics—in this work, post-compilation latency —across multiple devices . Current HW-NAS approaches jointly tackling (i-ii) typically rely on real-world performance measurements collected for a fixed target device, which are very expensive to collect as they require compilation of a large number of networks. Furthermore, current approaches to ensure cross-device performance rely on (a) strong assumptions circa analytical approximations of downstream hardware efficiency or (b) predictors approximating real-world measurements. Clearly, (a-b) do hinder practical applicability in risk-averse scenarios, as (a) makes assumptions that risk being too coarse for practical use cases, and the uncertainty circa (b) predictors is difficult to bound in practice. In this work, we propose a two-stage approach to HW-NAS mitigating the need to access information pre-deployment. With this, we elaborate a search-strategy focused on learning patterns for DNN HW-design for simulated versions of the target device, to then transfer to the real-device and collect few real-world measurements for adaptation. Initially, the controller learns to design DNNs under computational constraints on fictional synthetic devices. Then, we deploy the controller on the target device, where the controller is tasked with optimizing the architectures, directly using real-world measurements and thus completely avoiding inaccurate latency estimations. Critically, pre-training on synthetic devices drastically limits the number of networks tried at test-time, making real-world measurements a valid alternative for our method, as we learn to probe the right networks at test-time. By training on simulated devices, the controller acquires a model of how latency/accuracy patterns vary across devices. When deployed on the target device, the search strategy we develop can then transfer its prior knowledge to explore more effectively under the unknown target latency distribution. This allows us to completely abandon amortized estimation of latency values—obtained through coarse approximations or the use of predictors—and directly compile and measure real-world latency values on a handful of candidate networks, properly selected thanks to pre-training. To transfer solutions across simulated and real-world devices, we leverage recent advancements on Reinforcement Learning (RL), which we complement with the use of accuracy proxies to reduce the computational burden associated with RL-based NAS. Such training-free accuracy proxies are especially suitable in our setting because they are fast to compute and practical data-agnostic. In summary, our work makes the following key contribution: - [we use training-free metrics for NAS] - we develop a randomized RL training procedure for HW-NAS , crafting a controller which generalizes across diverse deployment devices at test time. This allows us to bypass inaccurate and expensive ways to estimate post-compilation latency, and rather use a very limited number of real-world measurements at test time. In summary, our work makes the following key contributions: - We reintroduce RL-based NAS, leveraging training-free metrics to estimate candidate architectures’ downstream performance. The use of training-free proxies enables rapid and efficient architecture evaluation without the computational overhead of full training. - We develop a randomized RL training procedure for HW-NAS , crafting a controller which generalizes across diverse deployment devices at test time. This allows us to bypass inaccurate and expensive ways to estimate post-compilation latency, rather learning to use a very limited number of real-world measurements (10) at test time. 2 Background Figure 1: Overview our method. HW-NAS across different hardware platforms is hindered by fundamental differences across devices, influencing the performance/efficiency tradeoff differently across different devices (A). Our method consists in a two-stage process where we first learn on a distribution of synthetic devices (B, 1), and then zero-shot transfer our learned policy to mulitple devices (B, 2). 2.1 Hardware-Aware Neural Architecture Search (HW-NAS) Neural Architecture Search (NAS) Designing Deep Neural Networks (DNNs) by hand is cumbersome and brittle process, originally carried out by highly-specialized human experts, and later automatized via Neural Architecture Search (NAS). In its seminal work, Zoph and Le, 2016 demonstrated the feasibility of designing DNNs automatically via RL, using post-training performance metrics as reward signal. While effective, ( Zoph and Le, 2016 ) ’s pioneering work most notably demanded a cluster of 800 GPU for training—a computational demand amply exceeding the capabilities of most institutions. Subsequent research introduced more computationally efficient approaches, including weight-sharing paradigms ( Brock et al., 2017 ) and gradient-based methods ( Liu et al., 2018 ) . In practice, it is common to search for architectures within cell-based searchspaces ( Zoph et al., 2018 ) , hereby reducing DNN design to identifying an optimal cell within a searchspace of all possible combinations, \mathcal{H} . To democratize NAS, the research community developed training-free metrics (TF metrics) ( Mellor et al., 2021 ; Jacot et al., 2018 ; Chen et al., 2021 ; Chen et al., 2023 ) , allowing to drastically reduce the and cost and resources needed to evaluate candidate architectures by scoring them at initialization, without any training ( Mellor et al., 2021 ; Tanaka et al., 2020 ; Cavagnero et al., 2023 ) . To search spaces of candidate architectures, Cavagnero et al., 2023 propose complemeting NASWOT ( Mellor et al., 2021 ) —measuring the number of "linear regions" formed by decision boundaries at initialization—with LogSynflow ( Cavagnero et al., 2023 ; Tanaka et al., 2020 ) —correlating the performance of a candidate design with the gradient flow of a loss function specifically designed to reflect the model’s capacity. Further,

Read the complete text