HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories
Eric Hedlin, Munawar Hayat, Fatih Porikli, Kwang Moo Yi, Shweta Mahajan · 2024
Read HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories by Eric Hedlin, Munawar Hayat, Fatih Porikli, Kwang Moo Yi, Shweta Mahajan (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17040.txt.
Introduction
Eric Hedlin and co-authors teach a network that builds other networks, without ever showing it a finished one, in a 2024 paper. Hypernetworks generate the weights of a task network on demand, powering personalization and generative models of neural behavior. Their training data is the problem: every sample needs a fully trained network to imitate, and training thousands of teachers is a project in itself.
The trick is to learn the journey, not the destination. Their Hypernetwork Field takes an extra input, the convergence state, and predicts the whole trajectory of a task network's weights as training proceeds. A built-in consistency law does the teaching: the predicted weights' gradients must match the task's true gradients at every state, and that constraint alone trains the field. Personalized image generation and 3D shape reconstruction matched methods that did use ground-truth weights.
Catalog record
| CREATOR | Eric Hedlin, Munawar Hayat, Fatih Porikli, Kwang Moo Yi, Shweta Mahajan |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17040
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HyperNet Fields: Efficiently Training Hypernetworks without Ground Truth by Learning Weight Trajectories Eric Hedlin, Munawar Hayat, Fatih Porikli, Kwang Moo Yi, Shweta Mahajan Abstract To efficiently adapt large models or to train generative models of neural representations, Hypernetworks have drawn interest. While hypernetworks work well, training them is cumbersome, and often requires ground truth optimized weights for each sample. However, obtaining each of these weights is a training problem of its own—one needs to train, e.g., adaptation weights or even an entire neural field for hypernetworks to regress to. In this work, we propose a method to train hypernetworks, without the need for any per-sample ground truth. Our key idea is to learn a Hypernetwork ‘Field’ and estimate the entire trajectory of network weight training instead of simply its converged state. In other words, we introduce an additional input to the Hypernetwork, the convergence state, which then makes it act as a neural field that models the entire convergence pathway of a task network. A critical benefit in doing so is that the gradient of the estimated weights at any convergence state must then match the gradients of the original task—this constraint alone is sufficient to train the Hypernetwork Field. We demonstrate the effectiveness of our method through the task of personalized image generation and 3D shape reconstruction from images and point clouds, demonstrating competitive results without any per-sample ground truth. Figure 1 : Teaser – We propose hypernetwork fields, where we learn the entire weight space trajectory instead of only the final state, which allows us to train without ever needing to know the final converged weights. Our method can be applied to any application of hypernetworks, including diffusion model personalization and modeling of 3D neural representations. 1 Introduction Various generative modeling tasks [ 40 ] require adaptation, thus data-dependent network weights, or in other words, model parameters being adjusted according to various circumstances (conditions and user input). These include personalized text-to-image generation where the weights of the underlying personalization model, such as DreamBooth [ 40 ] , are fine-tuned on the concept of interest and used to generate target images conditioned on input prompts. Another example is 3D modeling, where neural networks are used to model shapes implicitly with signed distance fields [ 30 ] or occupancy [ 29 ] . In both cases, a dedicated set of neural network weights is required for each personalization outcome or 3D shape. Thus, the applicability of these methods is highly limited by the training time required to obtain these weights. It is therefore critical to seek ways to amortize the per-sample optimizations into Hypernetworks [ 11 , 41 ] —networks that estimate network parameters. By ‘skipping ahead’ the tedious training processes involved in fine-tuning or shape regressing, they greatly speed up the entire process. Existing methods, however, have a critical shortcoming in that they supervise hypernetworks by directly matching their output to the precomputed, ground truth converged weights corresponding to the task-specific network for each sample in the dataset. Notably, prior to any training of the Hypernetwork, one must prepare converged task-specific network weights for every sample in the training set. This severely limits the capabilities of Hypernetwork methods when scaling to large datasets. As an example, to train HyperDreamBooth [ 41 ] , it requires 50 days of GPU time on a NVidia RTX 3090, 3 3 3 HyperDreamBooth [ 41 ] was trained on 15k DreamBooth weights, which each take 5 minutes to compute. to just prepare the training data alone. Furthermore, current methods assume a one-to-one mapping between each input sample and its corresponding optimized weights. This assumption is an oversimplification. Neural network optimization is stochastic with various equally plausible solutions. Therefore, enforcing a bijective mapping between the condition and the ultimately converged weights may limit the potential expressiveness of the hypernetwork, as we will demonstrate in Sec. 3 . To address these limitations, we instead propose to train a Hypernetwork Field that estimates the entire optimization trajectory of a neural network training, rather than focusing on a single final set of weights for each sample. Specifically, we introduce the convergence state as additional input to the hypernetwork, such that the weights of the task-specific network at a desired state of convergence can be ‘queried’. An interesting outcome of doing so is that the gradient of the estimated weights along this pathway must match the gradients of the original task-specific network, as they are both gradients pointing towards convergence. Hence, training of such hypernetwork fields can simply be done by matching the two gradients. By supervising the hypernetwork to match the gradient of the weights with respect to the optimization step, we avoid the need for precomputing target weights. This supervision strategy allows the hypernetwork to estimate a trajectory of parameters that, at each step, reflects a compatible state across all samples in the dataset. Ultimately, at inference time, the estimated parameters which correspond with the final timestep of the hypernetwork, represent a well-converged solution for each of the samples, but are now naturally discovered through training of the hypernetwork field itself. Notably, this single forward pass to estimating the converged parameters is all that is necessary for inference, hence no additional compute is required compared to conventional hypernetwork approaches [ 41 , 11 ] . Our framework is general and is applicable to all hypernetwork training scenarios. To demonstrate the effectiveness of our method, we apply our method to personalized image generation and 3D shape generation. We achieve competitive performance while reducing the training cost by \sim 4 times what would be required of conventional hypernetwork training. To summarize, our contributions are: • We propose hypernetwork fields, an approach to learn the entire optimization pathway of task-specific networks, which allows ground truth-free training without the need for precomputing the target weights; • our approach drastically reduces the compute required for training hypernetworks; • we validate our approach on two very different tasks and datasets—for personalized image generation we achieve results comparable to the state of the art with significantly less compute; and • we further explore the application of our method to 3D shape reconstruction from images an point clouds. 2 Related Work Hypernetworks. Early work on training hypernetworks backpropagates gradients through the task-specific network to the hypernetwork itself, as demonstrated by approaches [ 16 , 32 ] and subsequent explorations in continual learning [ 49 ] and neural architecture search [ 51 ] . These methods have shown that allowing gradient flow through both networks enables the hypernetwork to modify parameters for diverse tasks adaptively. However, more recent works have introduced approaches that involve precomputing weights for the task-specific network and directly supervise the hypernetwork to match them, eliminating the need for backpropagation through the task-specific network and improving training stability by alleviating issues with gradient flow. This supervision with precomputed weights has been shown to enhance gradient stability and training performance, as highlighted in frameworks like HyperDiffusion [ 11 ] , HyperFields [ 2 ] , and HyperDreamBooth [ 41 ] . However, this limits the size of the dataset to which this can be applied, as the precompute requirement is proportional to the number of samples in the dataset. Our method combines the