Instruction-Guided Autoregressive Neural Network Parameter Generation Soro Bedionita 1, Bruno Andreis 1, Song Chong 1, Sung Ju Hwang 1, 2] Abstract Learning to generate neural network parameters conditioned on task descriptions and architecture specifications is pivotal for advancing model adaptability and transfer learning. Existing methods—especially those based on diffusion models—suffer from limited scalability to large architectures, rigidity in handling varying network depths, and disjointed parameter generation that undermines inter-layer coherence. In this work, we propose IGPG (Instruction-Guided Parameter Generation), an autoregressive framework that unifies parameter synthesis across diverse tasks and architectures. IGPG leverages a VQ-VAE and an autoregressive model to generate neural network parameters, conditioned on task instructions, dataset, and architecture details. By autoregressively generating neural network weights’ tokens, IGPG ensures inter-layer coherence and enables efficient adaptation across models and datasets. Operating at the token level, IGPG effectively captures complex parameter distributions aggregated from a broad spectrum of pretrained models. Extensive experiments on multiple vision datasets demonstrate that IGPG consolidates diverse pretrained models into a single, flexible generative framework. The synthesized parameters achieve competitive or superior performance relative to state-of-the-art methods, especially in terms of scalability and efficiency when applied to large architectures. These results underscore IGPG’s potential as a powerful tool for pretrained weight retrieval, model selection, and rapid task-specific fine-tuning. 1 Introduction Deep neural networks have driven breakthroughs across domains—from image recognition ( Russakovsky et al., 2015 ; He et al., 2016 ) to natural language processing—leading to vast repositories of pretrained models ( Schürholt et al., 2022c ) available via platforms like Hugging Face 1 1 1 https://huggingface.co/ and libraries such as TIMM ( Wightman, 2019 ) . Despite their success, adapting these models to new tasks or datasets is challenging. It often requires manual intervention, extensive fine-tuning, and careful model selection. Prior work in transfer learning, meta-learning, and knowledge distillation ( Gou et al., 2021 ; Yang et al., 2021 ; Elsken et al., 2019 ) has predominantly focused on individual models, often overlooking the cross-task insights embedded in large-scale model collections. More recent efforts in hyper-representation learning ( Schürholt et al., 2021 ; Schürholt et al., 2022b ; Schürholt et al., 2022a ; Wang et al., 2024 ) have sought to learn distributions over network weights to enhance initialization. However, these methods are typically unconditional and limited to single-task scenarios, neglecting the potential benefits of incorporating pretrained dataset embeddings during training. While a few studies have explored task-specific parameter generation ( Soro et al., 2024 ) , there remains a significant gap in developing a unified and flexible solution. Furthermore, when applied to large architectures, these approaches tend to generate weight chunks without considering the relationships within each sampled layer, thereby limiting performance and increasing sampling time. To address these challenges, we introduce Instruction-Guided Parameter Generation (IGPG), a novel framework that integrates Vector Quantized Variational Autoencoders (VQ-VAE) with autoregressive modeling to generate neural network parameters conditioned on both task and architecture. IGPG jointly encodes three key elements: Task Representations : Using dataset embeddings or natural language instructions to capture target task semantics; Architecture Specifications : Explicitly representing network designs to enable cross-architecture parameter generation; and Inter-Layer Dependencies : Employing autoregressive modeling to preserve coherence across layers. This joint conditioning enables IGPG to efficiently synthesize coherent, task-optimized parameters, reducing reliance on extensive fine-tuning. Our contributions are summarized as follows: 1. Task-Conditioned Generation : We propose a mechanism for directly generating network parameters from natural language or dataset descriptors, offering intuitive task control. 2. Architecture-Agnostic Framework : Our method generates parameters across diverse architectures, leveraging knowledge from multiple pretrained models. 3. Autoregressive Coherence : By modeling layer-wise dependencies, IGPG ensures internally consistent parameters that accelerate convergence and enhance transfer performance. Extensive experiments demonstrate that IGPG compresses and transfers the collective knowledge of diverse pretrained models into a single generative framework, achieving competitive or superior performance on unseen tasks and scaling effectively to larger architectures (see Figure 1 ). Figure 1: Our approach integrates a VQ-VAE autoencoder ( \mathbf{E} – \mathbf{D} ) with a transformer prior. First, the VQ-VAE encodes vectorized network parameters (see Section 2.2 ), and then the transformer is trained on the resulting codebook (see Section 3 ). Additionally, prompts—including data, task, or architecture details—are processed using multimodal or language modeling techniques (see Section 3 ), with an example training simplified prompt template provided in Remark 1 . 2 Instruction-Guided Parameters Generation 2.1 Preliminary We introduce Instruction-Guided Parameter Generation (IGPG), a framework that learns the distribution of pretrained models to generate new weights on demand (see Figure 1 ). By capturing key feature patterns of high-performing networks, IGPG can generate specialized weights for both existing and novel tasks or dataset, reducing extensive retraining and accelerating model deployment in diverse computer vision scenarios. Our method begins with a set of pretrained models \{\theta_{i}\}_{i=1}^{N} and their corresponding datasets or task description \{D_{i}\}_{i=1}^{N} . We construct a model zoo by vectorizing each network’s parameters in one of two ways. In the layer-wise setting, each layer’s weights (including biases) are flattened into a single vector, yielding per-layer parameter samples from across all pretrained networks. Alternatively, in the architecture-wise setting, the flattened layer weights of each model are sequentially concatenated to form a single global parameter vector per model, preserving the original layer order. Both approaches produce uniform parameter representations that IGPG uses to learn a generative mapping, enabling the generation of dataset/task- and architecture-specific weights for efficient adaptation. We formalize our setup by defining \mathcal{D} as the space of possible datasets or tasks, \mathcal{A} as the space of neural architectures, and \Theta as the parameter space. Our generative mapping H operates in two phases: during training, H:\mathcal{D}\times\mathcal{A}\times\Theta\rightarrow\Theta ; during inference, H:\mathcal{D}\times\mathcal{A}\rightarrow\Theta . Thus, given a dataset D_{i} and architecture A_{i} , the trained H produces a tailored initialization \hat{\theta}_{i}=H(D_{i},A_{i}) . To enforce autoregressive parameter generation and capture layer-wise dependencies, we build IGPG based on a VQGAN structure combined with a transformer autoregressive prior. This design ensures coherent parameter generation by modeling dependencies between layers while leveraging the strengths of both architectures. 2.2 Neural Network Parameters Encoding with VQVAE We encode neural network parameters using a Gumbel Vector Quantized Variational Autoencoder (VQVAE) ( van den Oord et al., 2017 ) to generate discrete representations suitable for autoregressive modeling. For a parameter vector \Theta\in\mathbb{R}^{D} , we employ fixed-size chunking with chunk size K , padding \Theta to length D^{\prime}=\lceil\frac{D}{K}\rceil\times K and splitting it into n=\frac{D^{\prime}}{K} chunks for efficient processing. The VQVAE architecture consists of an encoder \mathbf{E} , decoder \mathbf{D} , and quantization module \mathbf{Q} with codebook \mathbf{e}=\{e_{1},...,e_{m}\} . For input parameters \theta , the encoder produces latent representations z=\mathbf{E}(\theta) , which are quantized using Gumbel-Softmax sampling: z_{q}=\sum_{j=1}^{m}y_{j}e_{j},\quad y_{j}=\frac{\exp((\log\pi_{j}+g_{j})/\tau)}{\sum_{i=1}^{m}\exp((\log\pi_{i}+g_{i})/\tau)} (1) where \pi_{j} are encoder logits, g_{j} are Gumbel noise samples, and \tau is the temperature parameter. The decoder reconstructs the input as \hat{\theta}=\mathbf{D}(z_{q}) . The model is optimized by minimizing: \mathcal{L}=\underbrace{\|\mathcal{M}\odot(\theta-\hat{\theta})\|_{2}^{2}}_{\text{reconstruction}}+\gamma\underbrace{\|z-\text{sg}[z_{q}]\|_{2}^{2}}_{\text{commitment}}+\beta\underbrace{\|\text{sg}[z]-z_{q}\|_{2}^{2}}_{\text{codebook}} (2) where \mathcal{M} masks padding values, \text{sg}[\cdot] denotes stop-gradient, and \{\beta,\gamma\} are balancing coefficients. This Gumbel-VQVAE formulation enables stochastic, differentiable quantization while preparing parameter vectors for subsequent autoregressive modeling using transformer architectures. 2.3 Autoregressive Modeling of Encoded Parameters We design an autoregressive framework that conditions parameter generation on both dataset content and network architecture. For a labeled dataset \mathcal{D} , we sample a balanced subset (e.g., five images per class) and embed each image using CLIP ( Radford et al., 2021 ) . Mean-pooling these embeddings yields the dataset-level vector \bar{e}_{\mathcal{D}} . To encode a network architecture \mathcal{A} , we convert its specifications into a standardized textual description \mathrm{desc}(\mathcal{A}) and process it with LLaMA-3-Instruct ( Dubey et al., 2024 ) , producing the architecture-level embedding e_{\mathcal{A}} . In the transformer’s forward pass, we concatenate \bar{e}_{\mathcal{D}} , e_{\mathcal{A}} , and the VQVAE codebook embeddings to form a unified representation for conditioning the autoregressive prior (e.g., GPT-2). Training Process : Following VQGAN ( Esser et al., 2020 ) , we employ a transformer-based prior (mini-GPT ( Radford et al., 2019 ) ) for conditional sampling. For each pretrained model, its encoded tokens are gathered into a single sequence representing the network. Our VQVAE is structured so that during training the autoregressive model can generate the full codebook in one pass based on next tokens prediction procedure where the context length is the length of the larger sequence vector. We train the GPT-based transformer to model the sequence likelihood and minimize the corresponding loss in a single formulation: \mathcal{L}_{\text{prior}}=\mathbb{E}_{s\sim p(s,e_{\mathcal{A}},\bar{e}_{\mathcal{D}})}\left[\log p(s\mid e_{\mathcal{A}},\bar{e}_{\mathcal{D}})\right],\quad\text{where}\quad p(s\mid e_{\mathcal{A}},\bar{e}_{\mathcal{D}})=\prod_{i}p(s_{i}\mid s_{N_{\text{max}} , we adopt chunk-wise autoregressive generation. Here, the model cannot generate all kl tokens at once without exceeding its maximum context size. Instead, we first generate an initial sequence \mathbf{s}^{(1)}\in\mathcal{V}^{N_{\text{max}}} via \mathcal{G}(e_{\mathcal{A}},e_{\mathcal{D}}) . We then iteratively generate additional token blocks \mathbf{s}^{(2)},\dots,\mathbf{s}^{(J)} , each conditioned on (e_{\mathcal{A}},e_{\mathcal{D}}) and a context window from the previously generated block, where J=\left\lceil\frac{kl}{N_{\text{max}}}\right\rceil. Concatenating all blocks yields \mathbf{s}_{\text{full}}\in\mathcal{V}^{N_{\text{max}}\cdot J} , which we truncate to the first kl tokens if necessary. Finally, we split \mathbf{s}_{\text{full}} into k segments of length l and decode each via \mathbf{D} to form the k chunks in \mathbb{R}^{K} . Flattening these chunks with \mathcal{F} produces the full parameter vector \theta_{\mathcal{A}} . Thus, by leveraging a chunked VQ-VAE representation and limiting each generation step to N_{\text{max}} tokens, we enable parameter generation for arbitrarily large architectures. Whenever kl\leq N_{\text{max}} , a single-step generation suffices; otherwise, we compose multiple chunks autoregressively. This design efficiently scales the generation process while maintaining the model’s capacity to represent high-dimensional parameter vectors. More details are provided in Algorithm 1 . 4 Experiments 4.1 Experimental Setup Implementation Details. All experiments are performed on a NVIDIA RTX V100 GPU. We train IGPG with AdamW and a linear learning rate schedule, starting from 1\times 10^{-4} . Datasets and Model Collection. We evaluate IGPG on a broad suite of pretrained models gathered from public repositories, covering diverse architectures and datasets. This setup enables a thorough examination of IGPG’s performance across different model scales, and data settings. The instructions used to guide the weights generation consist of text description of the architecture combined with dataset embeddings. Evaluation Protocol We evaluate IGPG through three primary experiments: 1. Comparison with existing methods on the pretrained model zoo from Schürholt et al. (2022a) 2. Generalization assessment across diverse ResNet architectures 3. Parameter generation efficiency evaluation on architectures with varying parameter counts Baselines We compare IGPG against three state-of-the-art approaches: Table 1: Comparison of weight initialization methods trained on pretrained from epochs 21–25. We compare: (1) training from scratch (tr. fr. scratch), (2) S_{\mathrm{KDE30}} ( Schürholt et al., 2022b ) , (3) SANE with KDE30 , (4) subsampled SANE SUB (aligned with IGPG), and (5) D2NWG ( Soro et al., 2024 ) . Ep. Method MNIST SVHN CIFAR-10 STL 0 tr. fr. scratch \sim 10 /% \sim 10 /% \sim 10 /% \sim 10 /% S_{KDE30} 68.6 \pm 6.7 54.5 \pm 5.9 n/a n/a SANE_{KDE30} 84.8 \pm 0.8 70.7 \pm 1.4 56.3 \pm 0.5 39.2 \pm 0.8 SANE_{SUB} 86.7 \pm 0.8 72.3 \pm 1.6 57.9 \pm 0.2 43.5 \pm 1.0 D2NWG 80.52 \pm 0.82 66.6 \pm 0.7 58.80 \pm 0.1 44.50 \pm 0.1 IGPG 83.20 \pm 0.01 67.10 \pm 0.4 58.3 \pm 0.1 44.41 \pm 0.1 1 tr. fr. scratch 20.6 \pm 1.6 19.4 \pm 0.6 37.2 \pm 1.4 21.3 \pm 1.6 S_{KDE30} 83.7 \pm 1.3 69.9 \pm 1.6 n/a n/a SANE_{KDE30} 85.5 \pm 0.8 71.3 \pm 1.4 58.2 \pm 0.2 43.5 \pm 0.7 SANE_{SUB} 87.5 \pm 0.6 73.3 \pm 1.4 59.1 \pm 0.3 44.3 \pm 1.0 D2NWG 87.8 \pm 0.4 73.6 \pm 1.3 59.2 \pm 0.3 44.8 \pm 0.2 IGPG 86.10 \pm 0.1 74.0 \pm 0.3 58.7 \pm 0.3 44.9 \pm 0.1 5 tr. fr. scratch 36.7 \pm 5.2 23.5 \pm 4.7 48.5 \pm 1.0 31.6 \pm 4.2 S_{KDE30} 92.4 \pm 0.7 57.3 \pm 12.4 n/a n/a SANE_{KDE30} 87.5 \pm 0.7 72.2 \pm 1.2 58.8 \pm 0.4 45.2 \pm 0.6 SANE_{SUB} 89.0 \pm 0.4 73.6 \pm 1.5 59.6 \pm 0.3 45.3 \pm 0.9 D2NWG 92.5 \pm 0.9 74.0 \pm 0.1 60.3 \pm 0.1 45.4 \pm 0.1 IGPG 91.7 \pm 0.8 74.5 \pm 0.5 60.3 \pm 0.1 45.7 \pm 0.1 25 tr. fr. scratch 83.3 \pm 2.6 66.7 \pm 8.5 57.2 \pm 0.8 44.0 \pm 1.0 S_{KDE30} 93.0 \pm 0.7 74.2 \pm 1.4 n/a n/a SANE_{KDE30} 92.0 \pm 0.3 74.7 \pm 0.8 60.2 \pm 0.6 48.4 \pm 0.5 SANE_{SUB} 92.3 \pm 0.4 75.1 \pm 1.0 61.2 \pm 0.1 48.0 \pm 0.4 D2NWG 96.2 \pm 0.3 75.7 \pm 0.5 64.1 \pm 1.0 48.7 \pm 0.5 IGPG 94.5 \pm 0.1 76.9 \pm 0.1 63.9 \pm 0.0 49.06 \pm 0.2 50 tr. fr. scratch 91.1 \pm 2.6 70.7 \pm 8.8 61.5 \pm 0.7 47.4 \pm 0.9 • Hyper-representations ( Schürholt et al., 2022a ) (S_{KDE} : A weights generation method that uses kernel density estimator(KDE) as prior. • SANE ( Sch”urholt et al., 2024 ) : An improved version of (S_{KDE} that uses weight tokenization with a KDE prior. • D2NWG ( Soro et al., 2024 ) : A diffusion-based approach to neural network weight generation. 4.2 Benchmarking on Tiny Model Zoo We evaluate IGPG on the Tiny Model Zoo dataset ( Schürholt et al., 2022c ) , which comprises compact CNNs trained on MNIST, SVHN, CIFAR-10, and STL-10. Specifically, we use a 2-layer CNN (2,464 parameters) for MNIST and SVHN, and a 3-layer CNN (10,853 parameters) for CIFAR-10 and STL-10. Following prior work, we draw pretrained weights from epochs 21–25 of a 50-epoch training schedule, with datasets split into training (70%), validation (15%), and test (15%). Unlike methods requiring separate models per dataset ( Schürholt et al., 2022a ; Sch”urholt et al., 2024 ) , IGPG learns a single generator that robustly handles all architectures and tasks. Task. We evaluate IGPG’s ability to generate neural network parameters that remain effective under both fine-tuning and transfer learning scenarios. In particular, we seek to confirm that IGPG’s synthesized weights are readily adaptable for fine-tuning scenarios. Results and Analysis. Table 1 shows that IGPG outperforms the sequential parameter generation method of Sch”urholt et al. (2024) , while matching the rapid convergence characteristic of state-of-the-art diffusion-based approaches. Crucially, IGPG preserves both zero-shot accuracy and fine-tuning performance when compared to previous works ( Sch”urholt et al., 2024 ; Schürholt et al., 2022a ) . This highlights IGPG’s capacity to generate robust initial weights that can be efficiently adapted, thereby accommodating multiple architectures and datasets within a single unified framework. 4.3 Transfer Learning and Fine-tuning on Unseen Datasets In this section, we train a single model on 30 diverse datasets from Meta-Album ( Ullah et al., 2022 ) , which span a broad range of class distributions. We then sample parameters for CIFAR-10 and Oxford Pets, thus evaluating how well a model pretrained on heterogeneous datasets adapts to unseen tasks. The training and target datasets are disjoint to ensure a fair assessment. Because we fix the architecture (a MobileNetV3 subnet from OFA) and pretrained with images of size 224 , we only encode dataset information rather than architectural details since the arhitecture is the same. Table 6 lists the training datasets. As shown in Figure 2 , our sampled models begin at a performance level on par with random initialization but achieve over 50% relative improvement within one epoch. This underscores the advantage of leveraging broad pretraining data for faster adaptation: although zero-shot performance may start near baseline, it quickly surpasses random initialization, reflecting the effectiveness of our method in generating meaningful parameters. 4.4 Cross-Architecture Benchmarking We evaluated our instruction-guided parameter generation on CIFAR-10 using 125 randomly sampled ResNet-56 variants spanning 200k–700k parameters, with block configurations from [4,4,4] to [8,8,8]. Each model was trained for 100 epochs, and we collected the last five epochs’ weights to form our training set. We set the maximum token-sequence length to 768 and trained a VQ-VAE to encode these parameters. Next, we employed a GPT-2 model, conditioned by an instruction template (preprocessed via LLaMA3-1-8B-Instruct), to generate the codebook tokens. We also experimented with fine-tuning larger language models on these codebooks, observing that while minor token mismatches (e.g. non-integers) can occur, the approach remains feasible. Figure 2: Transfer learning evaluation on novel datasets: CIFAR100, CIFAR10, Aircraft30, and PETS10 compared to random initialization. We then tested five ResNet architectures, including two in-distribution variants (directly sampled) and three out-of-distribution (ResNet-20, ResNet-56, and ResNet-110). Figure 3 compares IGPG against random initialization and CIFAR-100-pretrained weights. On in-distribution architectures, IGPG achieves accuracy on par with pretrained baselines. For out-of-distribution networks, our method attains up to 64% accuracy on ResNet-20 and 46% on both ResNet-56 and ResNet-110, outperforming the other baselines. These results highlight IGPG’s strong cross-architecture generalization without requiring additional fine-tuning, underscoring the potential of instruction-guided parameter generation in handling unseen network configurations. Figure 3: Performance evaluation with seen and unseen ResNet architectures on CIFAR-10 against models pretrained on CIFAR-100 and Random Initialization. (a) CIFAR10 (b) CIFAR100 Figure 4: Comparison of IGPG’s conditional sampled weight based initialization versus pretrained models across diverse architectures on CIFAR10 and CIFAR100 4.5 Handling Diverse Pretrained Models from Varied Datasets We further demonstrate IGPG’s versatility by encoding a broad set of architectures pretrained on datasets with varying numbers of classes (CIFAR-10 vs. CIFAR-100). We gather 19 publicly available models from GitHub 2 2 2 https://github.com/chenyaofo/pytorch-cifar-models spanning ResNet, ShuffleNet, MobileNet, and others. These architectures range from 0.27M to 27M parameters (over 100 \times difference), as shown in Figure 5 . Experimental Setup. We train a VQ-VAE with a chunk size of 2694256 parameters, each encoded into 64 tokens. For the largest models (roughly 10 chunks), this translates into a maximum sequence length of 640 tokens for our autoregressive model. We condition on both architecture descriptions and dataset encodings (via a CLIP image encoder using five images per class). Results and Significance. Figure 4 reports near-perfect Pearson correlations for both CIFAR-10 (0.9999) and CIFAR-100 (0.9991), suggesting that our generated parameters track closely with original pretrained weights. The regression lines for each dataset align closely with y=x , indicating comparable performance between IGPG-generated weights and their pretrained counterparts. These findings highlight IGPG’s capacity to learn from diverse architecture–dataset pairs and produce parameters that faithfully approximate original performance, pointing to its potential for guiding model selection and fast adaptation under dataset- or instruction-based constraints. 4.6 Extension to Diverse LoRA Parameters Generation To demonstrate that IGPG can learn distributions of diverse LoRA modules and improve downstream performance, we evaluated it on six standard computer vision benchmarks: Oxford-IIIT Pets, Stanford Cars, CIFAR-10, EuroSAT, the Describable Textures Dataset (DTD), and FGVC Aircraft. We began by fine-tuning a Vision Transformer (ViT-Base) with LoRA modules, following the procedure of Gao et al. (2024) . For each dataset, we retained the top five performing models and learned their parameter distributions. We then used a specialized version of IGPG to explore these distributions and generate novel LoRA parameters. Table 2 shows that our generated LoRA parameters deliver up to a 10% improvement compared to the baseline pretrained model ( Gao et al., 2024 ) , highlighting the efficacy of our distribution-learning approach in uncovering higher-performing LoRA configurations within the existing parameter space. Table 2: Performance evaluation on LoRA weights generation conditioned on the dataset Model Method # Trainable Parameters OxfordPets StanfordCars CIFAR10 DTD EuroSAT FGVC Average ViT-Base LP - 90.28 \pm 0.43 25.76 \pm 0.28 96.41 \pm 0.02 69.77 \pm 0.67 88.72 \pm 0.13 17.44 \pm 0.43 64.73 FF 85.8M 93.14 \pm 0.40 79.78 \pm 1.15 98.92 \pm 0.05 77.68 \pm 1.21 99.05 \pm 0.09 54.84 \pm 1.23 83.90 LoRA 581K 93.19 \pm 0.36 45.38 \pm 0.41 98.78 \pm 0.05 74.95 \pm 0.40 98.44 \pm 0.15 25.16 \pm 0.16 72.65 FourierFT ( Gao et al., 2024 ) 72K 93.21 \pm 0.26 46.11 \pm 0.24 98.58 \pm 0.07 75.09 \pm 0.37 98.29 \pm 0.04 27.51 \pm 0.64 73.13 IGPG 72K 92.84 \pm 0.30 57.67 \pm 0.52 98.45 \pm 0.95 88.74 \pm 0.08 98.70 \pm 0.23 36.63 \pm 0.11 78.83 4.7 Learning Distribution of models pretrained on Diverse Datasets To demonstrate that IGPG can learn distribution of multiple model pretrained on both large, medium and small scale dataset while maintaining a higher performance, we collect pretrained weights of ViT and Mobilenetv3 small pretrained on various datasets including ImageNet-1k(see Table 3 ) to train our method. Experiment Setup and Findings. We train VQ-VAE on combined weights from diverse datasets, and the transformer on its codebook using 5 samples per class and architecture descriptions as instructions, then evaluate both reconstruction and autoregressive sampling in-distribution. Table 3 reports the results as follows: Pretrained (%) : Accuracy of the original pretrained weights. VQ-VAE Reconstruction (%) : Accuracy after encoding and decoding the pretrained weights via VQ-VAE, showing the fidelity of our compressive representation. Best Sampled Weights (%) with IGPG : Accuracy of the best sample among five randomly drawn sequences from our trained model, conditioned on the respective dataset and architecture. For ViT-Small across CIFAR-10, CIFAR-100, CINIC-10, SVHN, and Tiny-ImageNet, we observe that both VQ-VAE reconstruction and IGPG sampled weights attain accuracy nearly indistinguishable from the original pretrained models. Moreover, the best samples often match or slightly exceed the baseline. On MobileNetV3-Small, sampled weights remain competitive with the pretrained baseline. These results confirm that IGPG preserves performance while offering the flexibility to sample diverse parameter configurations. Table 3: Learning distribution of combined ViT-Small (CIFAR-10, CIFAR-100, CINIC-10, SVHN, Tiny-ImageNet) and MobileNetV3-Small (ImageNet). We report model size (Parameters), Pretrained accuracy, VQVAE Reconstruction accuracy, and the best among five samples (Best Sampled Weights). Dataset Architecture Parameters Pretrained (%) VQVAE Reconstruction (%) IGPG(%) CIFAR-10 ViT-Small 2697610 93.67 93.67 93.70 CIFAR-100 ViT-Small 2714980 72.29 72.28 72.30 CINIC-10 ViT-Small 2697610 83.23 83.23 83.24 SVHN ViT-Small 2697610 97.74 97.74 97.76 Tiny-ImageNet ViT-Small 2771144 53.44 53.44 53.45 ImageNet MobileNetV3-Small 2542856 67.67 67.66 67.65 Additional results are presented in the Appendix B and Ablation in Appendix D . 5 Conclusion We introduced IGPG, an instruction-guided framework for neural network parameter generation that combines a VQVAE with autoregressive modeling. IGPG generates network parameters conditioned on task descriptions and architectural specifications. Experimental results demonstrate that IGPG achieves performance comparable to that of pretrained models while converging faster than random initialization. Furthermore, our approach effectively compresses large pretrained datasets and generalizes across diverse architectures, thereby advancing the fields of neural architecture search and transfer learning. Limitations : A key limitation of our method is its reliance on training with a large, diverse set of pretrained models—a comprehensive public repository of such models does not yet exist. However, this challenge is increasingly mitigated by the growing availability of models from repositories like Hugging Face and by efficient fine-tuning techniques such as LoRA. Appendix A Overview A.0.1 Vectorizing Neural Network Parameters To enable our generative mapping function H to learn from diverse pretrained models, we introduce a standardized parameter vectorization scheme that transforms weights and biases into a uniform vector representation. For a network with L layers, fully connected layers are vectorized by reshaping the weight matrix \theta^{(l)}\in\mathbb{R}^{d_{l-1}\times d_{l}} into \operatorname{vec}(\theta^{(l)})\in\mathbb{R}^{d_{l-1}d_{l}} and appending the bias b^{(l)}\in\mathbb{R}^{d_{l}} , yielding d_{l-1}d_{l}+d_{l} elements. Similarly, convolutional layers with kernel \theta^{(l)}\in\mathbb{R}^{k_{h}\times k_{w}\times c_{\text{in}}\times c_{\text{out}}} are flattened to \operatorname{vec}(\theta^{(l)})\in\mathbb{R}^{k_{h}k_{w}c_{\text{in}}c_{\text{out}}} and concatenated with the bias b^{(l)}\in\mathbb{R}^{c_{\text{out}}} to produce k_{h}k_{w}c_{\text{in}}c_{\text{out}}+c_{\text{out}} elements. We consider two aggregation strategies: an architecture-wise vectorization that concatenates all layers into a single vector \Theta=\bigoplus_{l=1}^{L}\left[\operatorname{vec}(\theta^{(l)})\oplus b^{(l)}\right] , and a layer-wise encoding that preserves each layer’s representation. A.1 Model Overview Our VQVAE model is a modified implementation based on the VQGAN codebase. Table 4 summarizes the architecture details, while Table 5 provides an overview of the generative model along with its parameter counts. We optimize the model using Adam with a learning rate of 1\times 10^{-4} and employ a cyclical temperature schedule for the Gumbel-Softmax, annealing the temperature from 1 to 1\times 10^{-4} . During both training and inference, when image dataset encoding is unnecessary, the model conditions solely on the architecture; otherwise, it leverages both architectural and dataset embeddings. We also illustrate an example template 5 for experimental results in Figure 3 , where only architecture embeddings are used since all architectures were pretrained on the same dataset. Index Name Type Parameters 0 encoder Encoder 197 M 1 decoder Decoder 198 M 2 loss VQLoss 0 3 quantize GumbelQuantize 132 K 4 quant_conv Conv2d 4.2 K 5 post_quant_conv Conv2d 4.2 K Table 4: Pretrained weights configuration for the VQVAE model. The table details the layer-wise parameters of the VQVAE model, highlighting its encoder, decoder, and quantization components optimized for downstream tasks. Index Name Type Parameters 0 first_stage_model GumbelVQNoDisc 48.5 M 1 cond_stage_model CondStage 524 K 2 ArchCond Identity 0 3 transformer GPT 27.8 M Table 5: Layer-wise configuration of the model, including the first-stage model, conditional stage model, and transformer components. The table specifies the type and parameter count of each component. This table shows the structure of the model used for the experiments in Section 4.2 Remark 1 . Instruction: You are a codebook generator. Input: Generate the full codebook of length 512 for ResNet with: num_blocks = [6, 7, 6]. Output:[995, 66, 750, 391, 809, 830,…]. Appendix B Additional Results and Tables Algorithm 1 Autoregressive Parameter Generation 1: Architecture encoding e_{\mathcal{A}} , dataset encoding e_{\mathcal{D}} , parameter length L , chunk size K 2: Architecture parameters \theta_{\mathcal{A}}\in\mathbb{R}^{L} 3: k\leftarrow\lceil L/K\rceil \triangleright Number of chunks 4: if kl\leq N_{\text{max}} then 5: \mathbf{s}\leftarrow\mathcal{G}(e_{\mathcal{A}},e_{\mathcal{D}}) \triangleright Single-pass generation 6: for i\leftarrow 1 to k do 7: \mathbf{s}_{i}\leftarrow\mathbf{s}_{(i-1)l:il} \triangleright Split into chunks 8: \theta_{i}\leftarrow\mathbf{D}(\mathbf{s}_{i}) \triangleright Decode chunks 9: end for 10: else 11: \mathbf{s}^{(1)}\leftarrow\mathcal{G}(e_{\mathcal{A}},e_{\mathcal{D}}) \triangleright Initial generation 12: for j\leftarrow 2 to \lceil kl/N_{\text{max}}\rceil do 13: \mathbf{s}^{(j)}\leftarrow\mathcal{G}(e_{\mathcal{A}},e_{\mathcal{D}},\mathbf{s}^{(j-1)}_{\text{ctx}}) 14: end for 15: \mathbf{s}_{\text{full}}\leftarrow\text{concat}(\mathbf{s}^{(1)},\ldots,\mathbf{s}^{(j)})_{1:kl} 16: for i\leftarrow 1 to k do 17: \mathbf{s}_{i}\leftarrow(\mathbf{s}_{\text{full}})_{(i-1)l:il} 18: \theta_{i}\leftarrow\mathbf{D}(\mathbf{s}_{i}) 19: end for 20: end if 21: \theta_{\mathcal{A}}\leftarrow\mathcal{F}([\theta_{1};\ldots;\theta_{k}]) \triangleright Flatten chunks 22: return \theta_{\mathcal{A}} Domain Original Dataset Name Number of Classes Large Animals Birds 315 Dogs 120 Animals with Attributes 50 Small Animals Plankton 102 Insects 2 102 Insects 117 Plants Flowers 102 PlantNet 25 Fungi 25 Plant Diseases Plant Village 38 Medicinal Leaf 26 PlantDoc 27 Microscopy Bacteria 33 PanNuke 19 Subcellular Human Protein 21 Remote Sensing RESISC 45 RSICB 45 RSD 43 Vehicles Cars 196 Airplanes 21 Boats 26 Manufacturing Textures 64 Textures DTD 47 Textures ALOT 250 Human Actions Sports 73 Stanford 40 Actions 40 MPII Human Pose 29 OCR Omniprint-MD-mix 706 Omniprint-MD-5-bis 706 Omniprint-MD-6 703 Table 6: Diverse pretrained datasets grouped by domain, showcasing their variety in classes for cross-dataset adaptation tasks. Figure 5: Parameters distribution of diverse architectures pretrained on CIFAR-10 and CIFAR100 all jointly encoded by IGPG In this section, we provide a comprehensive report of the experimental results such as those presented in section 4.5 . The primary objective of these results is to demonstrate the ability to encode diverse architectures across a variety of datasets while maintaining retrieval performance comparable to the pretrained models. Table 9 showcases representative results for different architectures, along with their corresponding parameter counts. The conditioning has been applied to both architectural configurations and datasets. Additionally, we include example configuration samples used for instruction encoding in Table 7 . Table 7: Examples of architecture configurations for various neural network families. Architecture Name Configuration vgg11_bn architecture: A, layers: [64, ’M’, 128, ’M’, 256, 256, ’M’, 512, 512, ’M’, 512, 512, ’M’], batch_norm: true mobilenetv2_x0_5 width_multiplier: 0.5, final_layer_channels: 640 repvgg_a0 width_multiplier: [0.75, 0.75, 0.75, 2.5] shufflenetv2_x0_5 stages_repeats: [4, 8, 4], stages_out_channels: [24, 48, 96, 192, 1024] resnet20 layers: [3, 3, 3] Appendix C Related Work Neural Network Parameter Generation Parameter generation for neural networks has evolved along two main trajectories: hypernetwork-based generation and generative hyper-representation learning. Hypernetwork approaches generate weights from scratch ( Stanley & Miikkulainen, 2002 ; Ratzlaff & Fuxin, 2019 ; Deutsch, 2018 ) , with recent advances in graph-hypernetworks ( Zhang et al., 2019 ; Knyazev et al., 2021 ) and transformer architectures ( Zhmoginov et al., 2022 ) . However, these methods primarily serve as initialization techniques requiring subsequent optimization. In contrast, generative hyper-representation learning focuses on modeling distributions of pretrained weights. Recent work has explored adaptive weight transfer ( Shu et al., 2021 ) , learning distribution of pretrained weights ( Schürholt et al., 2022b ) , and diffusion-based generation ( Peebles et al., 2022 ) which demonstrated promising results. Our current work extend these work to autoregressive generation preserving inter-layer relation in the network. Applications and Implications Weight generation, particularly from pretrained distributions, offers several key advantages in modern deep learning such as enabling efficient model compression and serving for large language models by enabling the generation of task-specific adaptations ( Huang et al., 2023 ; Addair & Rishi, 2024 ) or act as parameters retrieval-based systems similar to Zhao et al. (2024) . Second, it enhances transfer learning by enabling task-adaptive sampling ( Tang et al., 2024 ) , streamlining model adaptation. Appendix D Ablation Study Full Model vs Layer-Wise Sampling We compare two parameter generation approaches: full model-wise encoding and layer-wise encoding. In layer-wise encoding, we assemble parameters from each layer into separate training datasets, applying chunking and padding per layer. While both methods perform well on in-distribution parameters, layer-wise encoding shows superior generalization to novel architectures, suggesting better adaptability and robustness. Impact of Chunking Strategy We evaluate the effect of chunking network weights versus using complete weight vectors. Using pretrained weights from PyTorchHub 3 3 3 https://github.com/chenyaofo/pytorch-cifar-models , we assess ResNet56 and MobileNetV2 on CIFAR-10 and CIFAR-100. Results in Table 8 show that while chunking offers no significant advantage for medium-sized architectures, it becomes crucial for larger models where chunk-free approaches struggle to maintain performance. LLM-based Parameter Generation We investigate whether instruction-tuned LLMs can generate neural network parameters directly. Using LLaMA-3.2-1B-Instruct with LoRA fine-tuning and GPT-2 trained on sequence-to-sequence codebook generation, we find mixed results. While LLaMA-3.2-1B accurately generates initial tokens, it struggles with longer sequences. Similarly, GPT-2 with top-k sampling (k=1) successfully matches pretrained codebooks for small parameter sets but degrades significantly beyond 1024 dimensions. These results indicate that LLMs can generate VQVAE codebook parameters for small models, but scalability remains a significant challenge. Table 8: Performance comparison of conditional sampling with chunk-based and chunk-free approach. C-10 and C-100 refer to respectively CIFAR-10 and CIFAR-100 datasets pretrained chunk-free chunk-based models #Params top-1 top-5 top-1 top-5 top-1 top5 Datasets resnet-56 860026 94.37 99.83 94.40 \pm 0.01 99.82 \pm 0.00 94.19 \pm 0.01 99.87 \pm 0.00 C-10 Resnet-56 865876 72.63 91.94 72.45 \pm 0.01 91.96 \pm 0.01 72.41 \pm 0.2 91.80 \pm 0.03 C-100 mobilenetv2_x0_5 834324 70.88 91.72 71.14 \pm 0.01 92.48 \pm 0.04 70.93 \pm 0.00 92.55 \pm 0.02 C-100 D.1 Neural Network Codebook Generation Using Chat Models We conducted several attempts to generate neural network codebooks using large language models (LLMs) optimized for chat-based interactions. Our investigation with LLaMA-3.2-1B revealed significant challenges in adapting chat models to generate neural network parameters without pretraining on the specific codebook data. For instance, training GPT-2-small in a sequence-to-sequence fashion with the codebook, followed by instruction tuning, enabled the model to successfully generate a correct codebook in 96.6 However, simply applying LoRA tuning to an instruction-tuned LLaMA model resulted in the generation of short and inconsistent codebooks, often with mixed or incomplete outputs. These findings highlight the limitations of current chat models in directly producing neural network parameters without substantial fine-tuning or pretraining on task-specific data. In future work, we aim to explore this problem more comprehensively, focusing on improving parameter generation capabilities with chat models. Most of our experiments to date utilized GPT-2 variants, and we plan to expand this investigation with other LLM architectures. Table 9: Performance of weights generation from diverses models pretrained on CIFAR-10 and CIFAR-100 datasets Model CIFAR-10 CIFAR-100 IGPG Pretrained Params (M) IGPG Pretrained Params (M) MobileNetV2 (×0.5) 93.16 93.12 0.70 71.05 71.14 0.82 MobileNetV2 (×0.75) 94.16 94.08 1.37 74.15 74.06 1.48 MobileNetV2 (×1.0) 93.98 94.05 2.24 74.41 74.30 2.35 MobileNetV2 (×1.4) 94.26 94.22 4.33 75.97 76.29 4.50 Average (MobileNetV2) 93.89 93.87 2.16 73.90 73.95 2.29 RepVGG-A0 94.44 94.47 7.84 75.32 75.29 7.96 RepVGG-A1 94.93 94.93 12.82 76.51 76.45 12.94 RepVGG-A2 95.36 95.27 26.82 77.52 77.50 26.94 Average (RepVGG) 94.91 94.89 15.83 76.45 76.41 15.95 ResNet-20 92.47 92.59 0.27 68.92 68.84 0.28 ResNet-32 93.47 93.53 0.47 69.98 70.15 0.47 ResNet-44 93.92 94.01 0.66 71.39 71.64 0.67 ResNet-56 94.56 94.37 0.86 72.43 72.60 0.86 Average (ResNet) 93.61 93.63 0.57 70.68 70.81 0.57 ShuffleNetV2 (×0.5) 90.61 90.65 0.35 67.90 67.80 0.44 ShuffleNetV2 (×1.0) 92.75 93.30 1.26 72.55 72.59 1.36 ShuffleNetV2 (×1.5) 93.58 93.57 2.49 74.22 74.22 2.58 ShuffleNetV2 (×2.0) 94.02 93.99 5.37 75.39 75.49 5.55 Average (ShuffleNetV2) 92.74 92.88 2.37 72.52 72.53 2.48 VGG11-BN 92.85 92.78 9.76 70.76 70.78 9.80 VGG13-BN 94.01 94.00 9.94 74.40 74.62 9.99 VGG16-BN 94.19 94.15 15.25 73.98 74.00 15.30 VGG19-BN 94.09 93.91 20.57 73.82 73.87 20.61 Average (VGG) 93.79 93.71 13.88 73.24 73.32 13.93 Global Average (All Models) 93.74 93.76 6.96 73.36 73.42 7.00