Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discovery
Noureddine Kermiche · 2026
Read Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discovery by Noureddine Kermiche (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14375.txt.
Introduction
Kermiche proposes a modular design for sequential learning that walls each task off in its own expert module, curbing catastrophic forgetting. A distributed gatekeeper spots outliers to route new inputs, while teacher learning, student distillation, and router training run side by side during one local session, after which raw data is deleted. A tight-bottleneck autoencoder draws sharp borders in crowded latent space and flags returning tasks without spawning duplicate modules. Tests across vision and language tasks show strong retention with no loss of student fidelity.
Catalog record
| CREATOR | Noureddine Kermiche |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42201 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14375
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Modular Continual Learning via Zero-Leakage Reconstruction Routing and Autonomous Task Discovery Noureddine Kermiche Abstract Catastrophic forgetting remains a primary hurdle in sequential task learning for artificial neural networks. We propose a silicon-native modular architecture that achieves structural parameter isolation using Task-Specific Experts and a distributed, outlier-based Gatekeeper. Moving beyond traditional sequential consolidation, our framework utilizes a Simultaneous Pipeline where Teacher learning, Student distillation, and Router manifold acquisition occur in parallel while raw data is present in a localized training session. This approach ensures computational efficiency and complies with privacy mandates like GDPR by deleting raw data as soon as a task is learned. We demonstrate that a Tight-Bottleneck Autoencoder (TB-AE) can effectively distinguish semantically crowded manifolds in high-dimensional latent spaces, overcoming the posterior collapse inherent to standard variational methods. By establishing strict topological boundaries, our TB-AE resolves latent space crowding in 4096-D LLM embeddings to provide a robust, unsupervised novelty signal. Furthermore, we validate an Autonomous Retrieval mechanism that confidently identifies returning manifolds, enabling stable lifelong learning without redundant module instantiation. Empirical results demonstrate that our “Live Distillation” approach acts as a natural regularizer, achieving strong retention across computer vision and natural language processing domains without suffering a student fidelity gap. Keywords: Continual Learning, Catastrophic Forgetting, Knowledge Distillation, Mixture of Experts, Privacy-Preserving AI, Autoencoders, Zero-Leakage, Parameter-Efficient Fine-Tuning. 1 Introduction At the core of lifelong learning lies the stability-plasticity dilemma—the fundamental tension between a system’s plasticity (the ability to rapidly acquire and integrate new knowledge) and its stability (the ability to preserve prior knowledge without degradation). In standard artificial neural networks, high plasticity inevitably leads to the abrupt overwriting of previously optimized weights, a phenomenon widely known as catastrophic interference or catastrophic forgetting (CF) [1]. Historically, the most effective countermeasure against this forgetting has been Experience Replay, which interleaves historical data with new observations to preserve performance. However, modern industrial, financial, and medical deployments are governed by strict data privacy mandates, such as the European Union’s GDPR “Right to be Forgotten.” These regulations introduce a third, equally critical constraint to the traditional dilemma: zero-leakage . A viable, production-ready lifelong learning system must not only balance stability and plasticity, but it must do so without maintaining a persistent, legally compromising buffer of historical user data. More fundamentally, our approach is driven by a critical paradigm shift: the transition into the “post-monolith AI era” [13]. As articulated by recent industry analyses, the pursuit of a single, omniscient monolithic model that sequentially absorbs all domain knowledge is increasingly viewed as computationally intractable, legally precarious, and vulnerable to capacity saturation. The future of continuous adaptation points toward a decentralized ecosystem of specialized, task-specific modules—a “Cambrian Zoo” of models sharing routing APIs rather than overlapping parameters. In this paper, we propose a Modular Brain Architecture that facilitates complete parameter isolation. We permanently separate the learning engine (the Persistent Teacher) from the storage engine (the Frozen Student Experts), using reconstruction-based signatures (the Router) to autonomously navigate a growing, decentralized library of tasks. Unlike global classifiers that require joint training across all domains, our routing mechanism is entirely local and distributed. Each router operates as an independent outlier detector, trained solely on its present task and completely oblivious to the data of other tasks. Crucially, by utilizing a Simultaneous Pipeline, we consolidate knowledge in real-time while the raw data is still actively streaming in a localized session. This approach achieves weight stability for historical tasks, enables bounded forward-transfer plasticity where applicable, and provides zero-leakage compliance by enforcing the immediate and permanent purging of source data upon task boundary commitment. 2 Literature Survey: The Competitive Landscape Continual Learning (CL) methodologies are generally evaluated based on their ability to handle Forward Transfer, Backward Transfer, and Zero-Interference. We categorize the existing literature and contrast them with our proposed architecture. 2.1 Regularization-Based Methods Methods such as Elastic Weight Consolidation (EWC) [2] and Synaptic Intelligence (SI) utilize metrics like the Fisher Information Matrix to identify specific synaptic weights that are critical to previously learned tasks. By adding a penalty term to the loss function, these frameworks slow the modification of crucial weights during subsequent learning phases. - Limitation: These methods suffer from capacity saturation. As the sequence of tasks grows, the overlapping, competing constraints lead to “semantic blurring” or intransigence, where the network becomes too rigid to absorb new anomalies. - Our Distinction: By dynamically spawning and freezing independent modules, our architecture imposes no capacity bounds on previously learned tasks, yielding 0.0% backward interference regardless of sequence length. 2.2 Rehearsal and Generative Replay Experience Replay (ER) [3] maintains a subset or buffer of raw data from previous tasks to continually retrain the network alongside new data. Generative Replay [4] attempts to circumvent raw data storage by training a generative model (like a GAN or VAE) to synthesize fake historical samples for pseudo-rehearsal. - Limitation: ER fundamentally violates the “Zero-Leakage” mandate required for privacy compliance. Generative Replay, while technically buffer-free, is prone to mode collapse and cascading noise, where the “dreamt” historical data degrades over successive generations. - Our Distinction: Our framework achieves Zero-Leakage compliance via the Simultaneous Pipeline. Distillation occurs exclusively while the original data stream is active. Raw data is permanently purged upon task commitment, and no generative dreaming is required to prevent backward interference. 2.3 Distillation-Based Methods Learning without Forgetting (LwF) [4] utilizes knowledge distillation by passing new task data through the frozen previous state of the model to generate soft targets, encouraging the new model state to mimic the old one. - Limitation: LwF relies on shared representations. Over a long sequence of tasks, the shared feature extractor drifts. Because old tasks are evaluated on unfamiliar, novel data distributions, the soft targets become increasingly noisy and inaccurate, resulting in long-term degradation. - Our Distinction: Instead of sequential distillation on shifting data, we perform “Live Distillation” directly from a high-capacity Teacher to a task-specific Student exactly once, strictly on the true task manifold. The Student is then permanently isolated. 2.4 Structural and Architectural Methods Structural approaches dynamically expand the network. Progressive Neural Networks freeze old columns and instantiate new ones with lateral connections, while Mixture of Experts (MoE) [6] utilizes a global gating network to route inputs to various sub-networks. - Limitation: Standard MoE solves scaling issues but relies on a global, jointly-trained routing network. In a strict continual learning setting, the global router itself becomes a victim of catastrophic forgetting, misrouting data as new tasks overwhelm its classification boundaries. - Our Distinction: We employ decentralized, task-specific reconstruction routers. These routers evaluate inputs unilaterally and independently, eliminating the risk of a shared gating mechanism forgetting older tasks. 2.5 Task-Aware Information Routing (TAMiL) Recent frameworks like TAMiL (Task-Specific Attention Modules) [16] similarly propose using autoencoders to route information from a shared representation space. - Limitation: TAMiL explicitly relies on an experience replay buffer to stabilize its common representation space, which violates zero-leakage mandates. Furthermore, traditional undercomplete autoencoders suffer from posterior collapse in dense LLM embeddings. - Our Distinction: Our framework achieves zero-leakage compliance by bypassing replay entirely, utilizing the Simultaneous Pipeline. Additionally, we resolve high-dimensional collapse using the Tight-Bottleneck Autoencoder (TB-AE), and employ Contrastive Soft Routing during inference to gracefully handle ambiguous domain shifts. 2.6 Task-Specific Adaptation and Co-Training (AdaLL) Recent methods