NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces
Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, Miroslav Pajic · 2026
Read NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces by Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, Miroslav Pajic (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-00180.txt.
Introduction
Kim and colleagues present NNiT, a diffusion system that generates neural network weights without tying generation to fixed matrix sizes. It turns weight matrices into patches and models them as locally shaped fields. Graph HyperNetworks with a convolutional decoder align the weight space, creating the local links patch processing needs. On multilayer perceptrons, where permutation symmetry is sharpest, NNiTs generate working networks across architectures. On ManiSkill3 robot tasks, NNiT tops 85% success on unseen layouts while baselines fail, and the same pipeline works on MNIST classification.
Catalog record
| CREATOR | Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, Miroslav Pajic |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45102 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.00180
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
NNiT: Width-Agnostic Neural Network Generation with Structurally Aligned Weight Spaces Jiwoo Kim, Swarajh Mehta, Hao-Lun Hsu, Hyunwoo Ryu, Yudong Liu, Miroslav Pajic Abstract Generative modeling of neural network parameters is often tied to architectures because standard parameter representations rely on known weight-matrix dimensions. Generation is further complicated by permutation symmetries that allow networks to model similar input-output functions while having widely different, unaligned parameterizations. In this work, we introduce Neural Network Diffusion Transformers (NNiTs), which generate weights in a width-agnostic manner by tokenizing weight matrices into patches and modeling them as locally structured fields. We establish that Graph HyperNetworks (GHNs) with a convolutional neural network (CNN) decoder structurally align the weight space, creating the local correlation necessary for patch-based processing. Focusing on MLPs, where permutation symmetry is especially apparent, NNiT generates fully functional networks across a range of architectures. Our approach jointly models discrete architecture tokens and continuous weight patches within a single sequence model. On ManiSkill3 robotics tasks, NNiT achieves > 85 % >85\% success on architecture topologies unseen during training, while baseline approaches fail to generalize. 1 Introduction Figure 1 : Width-Agnostic Synthesis via Multimodal Tokenization . Unlike previous models, NNiT decouples functional logic from fixed matrix dimensions, allowing the zero-shot synthesis of optimal weights for architectural topologies entirely unseen during training. Diffusion Transformers have demonstrated scalability across various modalities such as computer vision image generation ( 33 ; 3 ; 53 ; 52 ; 9 ) , video generation ( 25 ; 13 ; 29 ; 5 ; 28 ; 10 ) , 3D generation ( 30 ; 51 ; 50 ) , and protein or molecule design ( 49 ; 15 ; 24 ; 20 ) . Recently, this generative paradigm has been applied to parameter synthesis, where models directly generate weights for functional neural networks ( 22 ; 42 ; 7 ) . These approaches aim to sample full neural networks from a learned distribution, bypassing the computational cost of traditional training. A central challenge in neural parameter synthesis is the permutation symmetry of network weights ( 31 ; 56 ) , where many distinct parameterizations correspond to the same function, so adjacent weights are spatially uncorrelated. To address this, recent approaches generate weights in a latent embedding ( 42 ; 38 ) , or apply explicit canonicalization methods ( 1 ) before flattening parameters into 1D vectors ( 39 ) . While these approaches can successfully transfer across varying depths, they remain fragile to changes in width. Once a weight matrix is collapsed into a fixed-dimensional vector, the generative prior becomes coupled to the matrix size seen during training. Even with alignment, changing the layer width changes token dimensionality and disrupts learned correspondence, preventing generalization to geometries unseen during training. To resolve this, we rethink the role of Graph HyperNetworks (GHNs) ( 55 ; 19 ) and apply them as both a data source generator and a mechanism for aligning weight space. GHNs propagate information over the architecture graph and generate layer parameters in fixed order, anchored at an input node with task features. In our implementation of the GHN, we use a CNN decoder, which imposes an explicit locality bias in the weight-space. This results in weights with consistent local spatial correlations across the population. On the other hand, standard Stochastic Gradient Descent (SGD) typically produces functionally equivalent solutions with arbitrary permutations resulting in unaligned parameter space. The GHN-induced parameter distribution thus provides the local structure needed for patch-based tokenization and width-agnostic transfer. Building on this alignment, we introduce NNiT, which formulates neural network synthesis as a single multimodal sequence modeling task. NNiT tokenizes aligned weight tensors into p\times p patches, replacing global vectors with spatially correlated tokens. This representation makes generation width-agnostic: widening a layer just corresponds to generating additional patches without changing the tokenization scheme. NNiT therefore learns a joint distribution over architectures and parameters , allowing generation of unseen architecture-weight pairs and the zero-shot synthesis of weights for user-specified structures. We evaluate NNiT on ManiSkill3 ( 43 ) using Multilayer Perceptron (MLP) policies for robotic control. Robotic manipulation is a strict evaluation metric where small weight errors may cause task failure, unlike vision tasks where artifacts could be tolerated. Moreover, generating robotic policies is an underexplored space where enabling joint weight and architecture generation opens new avenues for research, such as meta-learning across tasks or optimizing networks for specific hardware constraints for sim to real deployment. Since this domain naturally favors MLPs, it serves as an ideal benchmark for evaluating structural generalization. NNiT maintains high success rates on architectural widths unseen during training, while baselines fail. Our primary contributions are as follows: • We establish that GHNs align the weight space, reducing permutation-induced variability and enabling coordinate-based parameter field. • We introduce patch tokenization for weights, making generation width-agnostic and enabling zero-shot synthesis for unseen architecture topologies. • We introduce NNiT, a multimodal diffusion transformer that jointly models architectures and weights for both joint generation and conditional weight synthesis. 2 Related Work 2.1 Generative Neural Network Weight Models Parameter synthesis is increasingly formulated as a direct generative modeling problem, where neural weights are treated as structured data modalities ( 46 ; 21 ; 38 ; 7 ) . Recent approaches have successfully adapted diffusion ( 14 ; 41 ) and flow matching models ( 23 ) to approximate these complex parameter distributions. However, a fundamental constraint in this domain is the high dimensionality and inherent permutation symmetry of weight spaces. where arbitrary neuron ordering obscures geometric structure. To resolve this ambiguity, prior works have employed Variational Autoencoders (VAEs) to compress weights into permutation-invariant latent codes ( 38 ; 42 ) . Alternatively, methods such as SANE ( 39 ) utilize explicit canonicalization methods to align the weights into a shared coordinate frame. In this vein, recent studies ( 1 ; 35 ) validate that offline optimization can effectively collapse weight permutation symmetries. Notably, these approaches have demonstrated transferability across diverse architectural depths, successfully synthesizing parameters for Convolutional Neural Networks (CNNs) ( 39 ; 42 ) , yet they typically remain constrained to fixed-width architectures. 2.2 Diffusion Models Diffusion probabilistic models have established themselves as the standard backbone for generative tasks, demonstrating exceptional scalability across image ( 33 ; 3 ; 27 ; 47 ; 44 ) , video generation ( 25 ; 36 ; 16 ) , and robotics ( 17 ; 6 ; 37 ; 4 ; 34 ) . Diffusion Transformers (DiTs) advanced this paradigm by utilizing patch-based tokenization to process data as sequences. This structural innovation facilitates multimodal training by projecting diverse data types—such as audio, video, and text—into a unified representation space ( 25 ; 36 ; 16 ; 40 ; 18 ; 57 ; 8 ) . Most relevant to our framework, recent Flexible Diffusion Transformers (FiTs) ( 26 ; 48 ) have introduced mechanisms to process variable-resolution images. We leverage these primitives to decouple parameter synthesis from fixed tensor dimensions, treating neural weights as width-agnostic fields. 3 Preliminaries & Problem Formulation Figure 2 : NNiT Framework Overview. Left: Unified Generative Architecture. We formulate neural synthesis as a multimodal sequence task. Discrete architecture tokens (orange) and continuous weight matrices (blue) are unified into a single