Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition

Haris Khan, Shumail Asif, Sadia Asif · 2025

Read Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition by Haris Khan, Shumail Asif, Sadia Asif (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-20997.txt.

Introduction

Deployed AI models keep changing: new skills are added, old ones merged in, and sometimes a piece must be removed for legal reasons. Khan and co-authors propose Modular Delta Merging with Orthogonal Constraints, a framework for composing fine-tuned models cleanly. Each task's changes are stored as a delta from a shared base model. Before merging, every delta is projected into its own orthogonal subspace, so tasks cannot collide.

The projected pieces are then fused by optimization into one model that keeps each task's skill. The design supports adding new models over time and unmerging a piece on demand, for example to meet data rules. Extra stabilizers guard old knowledge during updates. On vision and language benchmarks, the method beat earlier baselines in accuracy, knowledge retention, and unmerge fidelity, while staying light on memory.

Catalog record

CREATORHaris Khan, Shumail Asif, Sadia Asif
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-20997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Modular Delta Merging with Orthogonal Constraints: A Scalable Framework for Continual and Reversible Model Composition Haris Khan, Shumail Asif, Sadia Asif Abstract In real-world machine learning deployments, models must be continually updated, composed, and when required, selectively undone. However, existing approaches to model merging and continual learning often suffer from task interference, catastrophic forgetting, or lack of reversibility. We propose Modular Delta Merging with Orthogonal Constraints (MDM-OC), a novel framework that enables scalable, interference-free, and reversible composition of fine-tuned models. Each task-specific model is encoded as a delta from a shared base and projected into an orthogonal subspace to eliminate conflict. These projected deltas are then merged via gradient-based optimization to form a unified model that retains performance across tasks. Our approach supports continual integration of new models, structured unmerging for compliance such as GDPR requirements, and model stability via elastic weight consolidation and synthetic replay. Extensive experiments on vision and natural language processing benchmarks demonstrate that MDM-OC outperforms prior baselines in accuracy, backward transfer, and unmerge fidelity, while remaining memory-efficient and computationally tractable. This framework offers a principled solution for modular and compliant AI system design. 1 Introduction Modern machine learning systems, particularly large-scale foundation models such as transformers [ 1 ] and vision transformers [ 2 ] , must continually evolve to meet the demands of dynamic environments, personalized services, and privacy-sensitive applications. A growing need has emerged for methods that can integrate multiple task-specific fine-tuned models into a unified system without retraining from scratch or compromising on generalization. This necessity is most evident in domains such as federated learning [ 3 ] , multi-agent intelligence systems [ 4 ] , and continual deployment of language or vision models in production environments [ 5 ] . The challenge of model composition has become increasingly complex as AI systems are deployed in regulated environments where compliance requirements such as the General Data Protection Regulation (GDPR) [ 6 ] mandate the ability to selectively remove or modify specific training influences. Traditional approaches to model merging through parameter interpolation or averaging often lead to catastrophic forgetting [ 7 ] , interference between tasks [ 8 ] , and the loss of critical knowledge acquired during task-specific fine-tuning. Moreover, these methods lack the reversibility required for compliance with data protection regulations that enforce the “right to be forgotten” [ 9 ] . Recent advances in model merging have explored various strategies including task arithmetic [ 10 ] , TIES-merging [ 11 ] , and adapter-based composition [ 12 ] . While these methods have shown promise in specific scenarios, they typically assume static model collections and do not address the fundamental challenges of continual integration and selective removal of model contributions. Furthermore, most existing approaches lack theoretical guarantees about interference reduction and provide limited scalability as the number of constituent models grows. This work addresses these limitations by proposing Modular Delta Merging with Orthogonal Constraints (MDM-OC), a novel framework that treats model merging as a structured, orthogonal projection problem in the parameter delta space. Rather than merging raw weights, our method extracts task-specific delta weights representing the minimal parameter changes induced by fine-tuning, and projects these deltas into orthogonal subspaces to ensure interference-free integration. The framework incorporates gradient-based optimization for fine-grained merging control and implements structured unmerging via algebraic operations in the projected delta space. Our approach is grounded in the mathematical principle that orthogonal vectors in parameter space can be composed and decomposed without mutual interference, providing theoretical foundations for both merging and unmerging operations. We extend this principle with practical considerations including numerical stability, computational efficiency, and preservation of model performance across diverse tasks. The contributions of this work are fourfold. First, we introduce a theoretically grounded delta-based orthogonal projection strategy that preserves task independence during merging while providing mathematical guarantees about interference reduction. Second, we propose a continual adaptation scheme that allows new models to be integrated or removed efficiently and reversibly, supporting dynamic model collections in production environments. Third, we combine our orthogonal merging framework with stability-preserving mechanisms including Elastic Weight Consolidation (EWC) [ 13 ] and synthetic replay techniques to maintain long-term model performance. Fourth, we provide comprehensive empirical validation demonstrating superior performance compared to state-of-the-art baselines across multiple domains and scales. This work bridges critical gaps in continual learning, parameter-efficient transfer learning, and modular AI systems, offering a practical and theoretically sound approach to composable, interpretable, and compliant AI architectures. The framework’s ability to support both forward composition and backward decomposition of model knowledge makes it particularly suitable for applications requiring auditability, compliance, and dynamic model management. 2 Related Work 2.1 Continual Learning and Catastrophic Forgetting Continual learning addresses catastrophic forgetting through various approaches. Regularization-based methods like Elastic Weight Consolidation (EWC) [ 13 ] use Fisher Information Matrix to protect important parameters, while Synaptic Intelligence [ 14 ] estimates parameter importance through accumulated gradients. Learning without Forgetting [ 15 ] employs knowledge distillation to preserve performance. These methods suffer from computational overhead, gradual performance degradation, and require access to previous task data. Gradient-based orthogonalization strategies ensure new updates don’t interfere with previous tasks. Orthogonal Gradient Descent (OGD) [ 16 ] projects gradients onto null spaces, while Gradient Episodic Memory (GEM) [ 17 ] and A-GEM [ 18 ] maintain episodic memories. Hebbian Learning based Orthogonal Projection [ 19 ] demonstrated orthogonality in spiking neural networks through lateral connections. However, these approaches operate at gradient level during training and don’t address post-hoc composition of independently trained models. 2.2 Model Merging and Parameter Composition Model merging evolved from simple parameter averaging in federated learning [ 3 ] to sophisticated composition strategies. Task Arithmetic [ 10 ] introduced task vectors as compositional units, while TIES-Merging [ 11 ] resolved conflicts through sign consensus and magnitude-based trimming. Model Soups [ 20 ] and Wise-FT [ 21 ] demonstrated that weight averaging can improve accuracy and robustness. Diverse Weight Averaging [ 22 ] incorporated diversity-based selection, and dataless knowledge fusion [ 23 ] explored merging without original training data. Parameter-efficient adapters provide new composition avenues. LoRA [ 9 ] captures adaptations through low-rank updates, while AdapterFusion [ 12 ] learns weighted combinations of adapters. Alignment-based strategies like model merging for knowledge editing [ 24 ] and continual learning in vision-language models [ 25 ] improved merging outcomes through parameter pre-alignment. Despite advances, most approaches lack formal interference guarantees and don’t support reversible operations. Our framework provides mathematical foundations for interference-free composition and algebraic reversibility through orthogonal projection in delta space. 2.3 Orthogonal Projections and Interference Reduction Orthogonal projections reduce interference by ensuring orthogonal vectors can be updated independently. Orthogonal Weight Modification (OWM) [ 26 ] maintained orthogonality in parameter updates for continual learning. Gradient Projection Memory (GPM) [ 19 ] projected updates orthogonal to previous task gradient subspaces. GopGAN projected orthogonal updates in generative models, while functional orthogonality analysis showed parameter orthogonality provides useful interference reduction approximations. Our work extends orthogonal projection concepts from gradient to parameter delta domain, enabling post-hoc composition of independently trained models without coordinated training or shared gradient information. 2.4 Knowledge Editing and Reversible Model Updates Large foundation models require knowledge editing and reversible updates. ROME [ 24 ] and MEMIT [ 27 ] demonstrated targeted knowledge

Read the complete text