Prototype Augmented Hypernetworks for Continual Learning

Neil De La Fuente, Maria Pilligua, Daniel Vidal, Albin Soutiff, Cecilia Curreli, Daniel Cremers, Andrey Barsky, [1.5mm] Computer Vision Center · 2025

Read Prototype Augmented Hypernetworks for Continual Learning by Neil De La Fuente, Maria Pilligua, Daniel Vidal, Albin Soutiff, Cecilia Curreli, Daniel Cremers, Andrey Barsky, [1.5mm] Computer Vision Center (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-07450.txt.

Introduction

Neil De La Fuente, Maria Pilligua, and co-authors give a hyper net with task marks that learns task chains with scant loss, in a 2025 paper. Chain learn must add new tasks sans wipe of old skills. Weight shifts for new tasks oft stomp old task weights in plain nets.

One hyper net spins up task heads on call from taught task marks. Two teach back losses keep old scores and marks in line as new tasks land. Tests on split shot sets hit top marks with tiny loss shares. No saved shots or old heads are kept in the frame. Both loss shares stay in low ones in the runs.

Catalog record

CREATORNeil De La Fuente, Maria Pilligua, Daniel Vidal, Albin Soutiff, Cecilia Curreli, Daniel Cremers, Andrey Barsky, [1.5mm] Computer Vision Center
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-07450

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Prototype Augmented Hypernetworks for Continual Learning Neil De La Fuente, Maria Pilligua, Daniel Vidal, Albin Soutiff, Cecilia Curreli, Daniel Cremers, Andrey Barsky, [1.5mm] Computer Vision Center Abstract Continual learning (CL) aims to learn a sequence of tasks without forgetting prior knowledge, but gradient updates for a new task often overwrite the weights learned earlier, causing catastrophic forgetting (CF). We propose Prototype-Augmented Hypernetworks ( P ​ A ​ H PAH ), a framework where a single hypernetwork, conditioned on learnable task prototypes, dynamically generates task-specific classifier heads on demand. To mitigate forgetting, P ​ A ​ H PAH combines cross-entropy with dual distillation losses, one to align logits and another to align prototypes, ensuring stable feature representations across tasks. Evaluations on Split-CIFAR100 and TinyImageNet demonstrate that PAH achieves state-of-the-art performance, reaching 74.5% and 63.7% accuracy with only 1.7% and 4.4% forgetting, respectively, surpassing prior methods without storing samples or heads. 1 Introduction Continual learning (CL) enables models to learn sequential tasks without forgetting prior knowledge, but catastrophic forgetting (CF) remains a major obstacle. Addressing CF is crucial for developing AI systems that can adapt to evolving environments, such as in robotics, autonomous driving, or personalized healthcare [ 24 ] . Recent work suggests CF primarily affects final classification layers due to representation drift [ 4 , 19 ] . While some class-incremental methods address drift [ 26 , 6 , 17 ] , task-incremental learning (TIL) often relies on fixed task-specific classifiers, failing to adapt them to evolving features. To address this classifier adaptation challenge in TIL, we introduce Prototype-Augmented Hypernetworks ( PAH ). PAH employs a shared hypernetwork, conditioned on learnable task prototypes , to dynamically generate task-specific classifier heads. This approach, combined with knowledge distillation [ 15 ] to preserve representations, effectively mitigates forgetting without requiring stored classifier heads. Our core contributions include a) an effective prototype-based task embedding mechanism, b) its integration with hypernetworks for dynamic head generation, and c) demonstrating state-of-the-art performance with near-zero forgetting on standard CL benchmarks. 2 Related Work Continual learning aims to learn sequential tasks without catastrophic forgetting [ 18 , 5 ] . Major approaches include regularization methods that penalize changes to important weights (EWC [ 10 ] ), rehearsal methods that replay stored or generated data (DER [ 1 ] , GCR [ 23 ] ), and architecture-based methods that allocate new parameters per task (PNN [ 21 ] ). These strategies often involve trade-offs regarding memory usage, computational overhead, or plasticity limits. Our work intersects hypernetworks and prototype‑based methods for continual learning. Hypernetworks [ 7 ] generate parameters for a target network and have been adapted to CL by conditioning on task identity to produce either full‑network weights or selected layers [ 25 , 8 ] . PAH keeps the same generator idea but feeds it learnable task prototypes to dynamically generate weights only for the task-specific classifier head, avoiding the need to store separate heads. The prototypes–compact class exemplars–have a long history in few‑shot learning [ 22 ] and as anchors for replay or nearest‑neighbour decisions in CL [ 20 , 9 ] . In PAH prototypes act instead as a task embedding that conditions the hypernetwork; while knowledge distillation [ 15 ] limits forgetting. The approach differs from Prototype Augmentation and Self‑Supervision (PASS) [ 28 ] , which keeps one prototype per class and feeds those fixed vectors directly to the classifier. PAH stores no classifier heads and never sends prototypes directly to softmax at test time. Figure 1 : Overview of the Prototype-Augmented Hypernetworks ( PAH ) architecture at task k. The hypernetwork generates task-specific classifier weights conditioned on learnable task prototypes e_{k} , which encode task-specific information. This design eliminates the need to store separate classifier heads for each task and effectively addresses catastrophic forgetting through knowledge distillation. 3 Problem Statement Formally, we consider a continual multitask learning problem in the task incremental paradigm. The model is presented with a sequence of tasks \{T_{1},T_{2},...,T_{K}\} , where each task T_{k} is associated with a dataset \mathcal{D}_{k}=\{(x^{i}_{k},y^{i}_{k})\}_{i=1}^{N_{i}} , where x^{i}_{k} and y^{i}_{k} denote the input data and labels respectively for the i -th sample in the k -th task. Each task consists of a set of classes unique to that task, and the objective is to learn a model that can perform well on all tasks seen so far, without revisiting previous data. Crucially, the task identity is available during training and inference. 4 Methodology PAH relies on three main components: (a) a main network, (b) a hypernetwork, and (c) learnable task embeddings. a) Main Network The main network comprises a backbone and a task-specific classification head. The backbone extracts features from incoming data, while the classification head assigns class labels based on the current task. Unlike traditional approaches that store a separate head per task, PAH does not require the explicit storage of task-specific head’s weights. Instead, they are generated on demand by the hypernetwork. Formally, given an input x_{k} from task T_{k} , the main network processes it as follows: f_{k}(x_{k}\,|\,k)=\tau(\beta_{\theta}(x_{k})\,|\,\theta_{k}^{\tau}) (1) where \beta_{\theta} represents the backbone, and \tau is the classification head parameterized by task-specific weights \theta_{k}^{\tau} . b) Hypernetwork A hypernetwork \Upsilon is a net that outputs the weights of another network. In this case, it takes a task embedding e_{k} as input and outputs task-specific classifier weights: \theta_{k}^{\tau}=\Upsilon(e_{k}) (2) c) Task Embedding The task embedding is the input that the hypernet receives, and it should help the hypernet identify the task and generate the correct weights for it. In PAH, it is formed by concatenating learned 2D prototypes (e.g., a compact 10 × 10 feature grid that preserves spatial layout), where each prototype serves as a representative feature for a class within a task. Formally, the task embedding is built as: e_{k}=\text{Flatten}({p^{1}_{k},p^{2}_{k},...,p^{C}_{k}}) (3) These prototypes provide the hypernetwork with a compact yet rich encoding, enabling it to generate more precise task-specific classifier weights. Unlike fixed embeddings, prototypes are dynamically optimized during training, learning to capture class-specific feature distributions, and ensuring alignment with evolving backbone representations. This approach allows the hypernet to leverage task- and class-specific information, improving adaptability while reducing the need to store separate classifier heads. Unless noted, each prototype starts as a random training image from its class, resized to 10\times 10 and normalized; we call this semantic initialization . 4.1 Loss Functions We use several loss terms tailored to address both classification accuracy and the stability-plasticity trade-off. In addition to the main image-based losses ( L_{hm} and L_{sm} ), we use prototype-based losses to refine task embeddings ( L_{sp} ). Hard Loss Main ( L_{hm} ) : This term uses the standard cross-entropy loss over the real input images to classify: L_{hm}=-\sum_{c=1}^{C}y_{c}\log\hat{y}_{c} (4) Here, y_{c} is the ground-truth label for class c , and \hat{y}_{c} is the model’s predicted probability. Minimizing L_{hm} ensures accurate classification on the current task. As illustrated in Figure 1 , this loss backpropagates through multiple components of the network, updating the backbone, hypernetwork, and task embeddings collectively. Soft Loss Main ( L_{sm} ) : This KL-divergence term preserves knowledge from previous tasks. When learning task T_{k} , for an input x_{k} from T_{k} : L_{sm}=\frac{1}{k\text{-}1}\sum_{j=1}^{k-1}\text{KL}(f_{k-1}(x_{k}\,|\,j)\;\|\;f_{k}(x_{k}\,|\,j)) (5) Here, f_{k-1}(x_{k}\,|\,j) is the output distribution of the model

Read the complete text