Arch-VQ: Discrete Architecture Representation Learning with Autoregressive Priors

Deshani Geethika Poddenige, Sachith Seneviratne Asela Hevapathige, Damith Senanayake, Mahesan Niranjan, PN Suganthan, Saman Halgamuge (✉) · 2025

Read Arch-VQ: Discrete Architecture Representation Learning with Autoregressive Priors by Deshani Geethika Poddenige, Sachith Seneviratne Asela Hevapathige, Damith Senanayake, Mahesan Niranjan, PN Suganthan, Saman Halgamuge (✉) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22063.txt.

Introduction

Searching the space of neural network designs needs a good way to write a design down as numbers a model can learn from. Deshani Geethika Poddenige and co-authors argue the usual smooth encodings fight the subject: architectures are discrete, so their codes should be too. Their framework, Arch-VQ, learns a codebook of discrete tokens for network designs.

A transformer then learns the grammar of those token strings, so new designs can be sampled token by token. Across three standard design benchmarks, the share of valid, unique generated networks rose sharply over the best rivals, more than doubling on one. The discrete codes also helped predictors that guess a design's accuracy. The authors say matching the code to the discrete nature of the problem is the whole trick.

Catalog record

CREATORDeshani Geethika Poddenige, Sachith Seneviratne Asela Hevapathige, Damith Senanayake, Mahesan Niranjan, PN Suganthan, Saman Halgamuge (✉)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22063

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Arch-VQ: Discrete Architecture Representation Learning with Autoregressive Priors Deshani Geethika Poddenige, Sachith Seneviratne Asela Hevapathige, Damith Senanayake, Mahesan Niranjan, PN Suganthan, Saman Halgamuge (✉) Abstract Existing neural architecture representation learning methods focus on continuous representation learning, typically using Variational Autoencoders (VAEs) to map discrete architectures onto a continuous Gaussian distribution. However, sampling from these spaces often leads to a high percentage of invalid or duplicate neural architectures, likely due to the unnatural mapping of inherently discrete architectural space onto a continuous space. In this work, we revisit architecture representation learning from a fundamentally discrete perspective. We propose Arch-VQ, a framework that learns a discrete latent space of neural architectures using a Vector-Quantized Variational Autoencoder (VQ-VAE), and models the latent prior with an autoregressive transformer. This formulation yields discrete architecture representations that are better aligned with the underlying search space while decoupling representation learning from prior modeling. Across NASBench-101, NASBench-201, and DARTS search spaces, Arch-VQ improves the quality of generated architectures, increasing the rate of valid and unique generations by 22%, 26%, and 135%, respectively, over state-of-the-art baselines. We further show that modeling discrete embeddings autoregressively enhances downstream neural predictor performance, establishing the practical utility of this discrete formulation. 1 Introduction Figure 1 : Illustration of the proposed Arch-VQ framework. (1) Step 1: A VQ-VAE is trained to learn a discrete latent space of neural architectures. Each architecture, represented by its adjacency matrix and one-hot node features, is passed through the encoder E to obtain the latent representation Z_{e} . The vector quantization module then maps Z_{e} to its nearest codebook entry, producing the discrete indices Z and quantized latent vector Z_{q} . The decoder D reconstructs Z_{q} back into the original architecture representation. (2) Step 2: The discrete index sequence Z is augmented with start and end tokens and used to train a transformer with a next-token prediction objective. (3) Step 3: The trained transformer generates new architecture sequences autoregressively, beginning from the start token. Unsupervised representation learning has driven transformative paradigm shifts across domains such as computer vision [ 20 , 7 ] , language representations [ 15 , 4 , 16 ] , and audio processing [ 8 , 3 ] . Yet its potential remains relatively underexplored in the neural architecture domain. Existing work on neural architecture representation learning has emerged in the context of Neural Architecture Search (NAS), where early methods operated directly in the original encoding space of neural architectures, typically represented using adjacency matrix and a list of operations [ 28 , 25 , 18 , 17 ] . However, the lack of explicit structural organization and poor scalability motivated a shift toward learning continuous architecture embeddings and conducting search in a latent space. A common strategy in this line of work is to learn representations jointly with optimization in a supervised manner [ 14 , 11 ] , prioritizing the NAS objective rather than modeling the intrinsic distribution of architectures. Existing unsupervised representation learning approaches are most notably based on Variational Autoencoders (VAE) [ 27 , 23 , 12 ] . Despite their empirical utility, continuous latent models like VAEs impose an implicit assumption: that the architecture distribution can be adequately approximated by a continuous Gaussian prior. Neural architectures, however, are discrete combinatorial objects defined by graph topology and categorical operations. Due to this fundamental mismatch, sampling from a continuous latent space and decoding back to discrete graphs often produces a high percentage of invalid [ 23 ] or duplicate samples [ 12 , 13 , 27 ] . As a result, large numbers of samples must be generated to obtain a decent proportion of valid and distinct architectures, increasing computational cost and limiting the latent space reliability. This mismatch points to a fundamental gap in the literature: no existing unsupervised method has explored a representation space that is natively discrete and aligned with the combinatorial structure of neural architectures. Discrete representation learning has proven effective in other discrete modalities [ 21 , 6 , 19 , 24 ] , where Vector-Quantized Variational Autoencoders (VQ-VAEs) learn codebook-based latent embeddings and decouple representation learning from prior modeling. VQ-VAE offers several key advantages over VAEs, such as: (i) they construct a discrete latent space by mapping each input to a set of discrete latent codes; and (ii) instead of relying on a fixed Gaussian prior, they learn the prior, enabling flexibility to pair with sophisticated autoregressive models for prior modeling. In this paper, we extend this paradigm to neural architectures and introduce Arch-VQ, a framework for unsupervised discrete representation learning of architecture graphs. To the best of our knowledge, Arch-VQ is the first framework to learn a discrete latent space of neural architectures via VQ-VAE, and the first to model the resulting discrete codes with an autoregressive transformer prior — two design choices that are individually motivated and jointly critical. First, we construct a discrete latent space using a VQ-VAE built upon graph encoders, mapping each architecture to a sequence of codebook indices. Second, we learn a prior over these discrete sequences using a decoder-only transformer trained with causal language modeling. This decoupling of representation learning from prior modeling is a key architectural distinction from prior VAE-based methods, which conflate both objectives within a single Gaussian framework. We evaluate Arch-VQ on NASBench-101 [ 26 ] , NASBench-201 [ 5 ] , and DARTS [ 11 ] search spaces against unsupervised VAE-based and unconditional generative baselines. Across all benchmarks, Arch-VQ significantly improves architecture generation quality, increasing the proportion of valid and unique samples by 22%, 26%, and 135%, respectively, compared to state-of-the-art methods, with the largest gains observed on the more complex DARTS search space. Moreover, by varying the sampling temperature, Arch-VQ provides a controllable balance between fidelity and exploration, yielding 20% and 135% improvements in valid and novel generations on NASBench-201 and DARTS, respectively. We further show that the learned discrete sequential representation improves downstream neural performance prediction. Our contributions are summarized as follows: • We propose Arch-VQ, a novel framework for unsupervised discrete representation learning of neural architectures using VQ-VAE — the first method to apply discrete latent modeling to the neural architecture domain. • We introduce autoregressive prior modeling over discrete architecture codes, enabling efficient architecture generation. This combination of VQ-VAE and autoregressive prior is a new formulation that no prior NAS representation learning work has explored. • We empirically demonstrate improvements in the quality of latent space over state-of-the-art baselines across multiple neural architecture benchmarks, with particularly large gains in the DARTS search space. • We show that modeling architectures as discrete sequences yields better neural performance prediction, demonstrating that the representational benefits of discretization extend beyond generation to downstream tasks. 2 Related Work Neural architecture representation learning Neural architecture representation learning has mainly developed within the NAS literature, where most methods rely on performance supervision, which shapes the representation to a specific optimization objective [ 14 ] . Our focus is on unsupervised architecture representation learning, where the goal is to model the structure of architecture spaces without performance labels. Existing unsupervised methods, including Arch2vec, D-VAE, and SVGe, typically adopt VAE-style continuous latent spaces for discrete architecture graphs [ 23 , 27 , 12 ] . This design introduces a mismatch

Read the complete text