AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning

Guransh Singh · 2026

Read AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning by Guransh Singh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-16067.txt.

Introduction

Guransh Singh presents AEGIS, a buffer-free orthogonal gradient projection framework for vision-language fine-tuning. Fine-tuning creates a stability-plasticity dilemma: flow-matching experts send regression gradients into cross-entropy backbones, harming vision. AEGIS measures activation stats from pre-training data as an anchor, then a Wasserstein-2 transport penalty creates an anchor-repair gradient. A backward pass projects updates aside during conflicts. The method sheds only 0.62 percent of energy, halting drift. On LIBERO, it preserves visual question answering and holdout loss while matching action convergence, without buffers.

Catalog record

CREATORGuransh Singh
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54087
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.16067

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AEGIS: Anchor-Enforced Gradient Isolation for Knowledge-Preserving Vision-Language-Action Fine-Tuning Guransh Singh Abstract Adapting pre-trained vision-language models (VLMs) for robotic control requires injecting high-magnitude continuous gradients from a flow-matching action expert into a backbone trained exclusively with cross-entropy. This cross-modal gradient asymmetry —the spectral dimensionality mismatch between low-rank MSE regression gradients and the high-dimensional semantic manifold sculpted by CE pre-training, causes rapid, severe erosion of the VLM’s visual-question-answering (VQA) capability. Industry-standard defences either sever the gradient pathway entirely via stop-gradient, discarding the rich continuous supervision, or restrict parameter capacity through low-rank adapters (LoRA) that constrain the rank of updates but not their direction , and thus still overwrite the pre-trained manifold. We introduce AEGIS ( A nchor- E nforced G radient I solation S ystem): a buffer-free, layer-wise orthogonal gradient projection framework that enables direct continuous MSE learning while preserving the pre-trained VQA manifold—without any co-training data or replay buffer. AEGIS pre-computes a static Gaussian reference anchor from masked VQA forward passes across all transformer layers, then at each training step constructs a Wasserstein- 2 2 transport penalty that generates an anchor-restoration gradient. A sequential dual-backward decomposes the task and anchor gradients; for each transformer layer, AEGIS applies a single Gram–Schmidt orthogonal projection that bends the task gradient away from the destructive direction while preserving its constructive content. The projection sheds less than 1 % 1\% of gradient energy on average, yet eliminates the cumulative activation drift that drives severe forgetting. 1 Introduction Vision-Language-Action (VLA) models adapt pre-trained VLMs for robotic control by fine-tuning on action-labelled demonstration data ( 3 ; 2 ) . The promise is compelling: a single foundation model that reasons about language commands, understands visual scenes, and produces continuous motor actions. The reality is precarious: the VLM backbone contains billions of parameters whose loss-landscape geometry has been sculpted by trillions of cross-entropy tokens; action fine-tuning injects a qualitatively alien training signal that overwrites these representations destructively. The cross-modal gradient asymmetry. A VLM pre-trained with cross-entropy (CE) develops an intricate network of delicate visual-semantic parameters. Conversely, a flow-matching action expert ( 2 ) produces high-magnitude, low-rank MSE gradients targeting a narrow physical subspace. When these high-magnitude MSE gradients backpropagate into the VLM, standard optimisers blast this concentrated robotic update indiscriminately across the network, destructively overwriting the semantic representations. We term this fundamental geometric mismatch cross-modal gradient asymmetry . In our experiments, naive fine-tuning that allows direct MSE gradient flow causes a significant, steady degradation in VQA holdout loss within 1{,}500 steps ( Figure 3 ), severely eroding the VLM’s pre-trained visual reasoning capability. The co-training tax. Current industrial VLA systems employ two workarounds to mitigate this problem, both of which impose substantial costs: 1. Stop-gradient routing ( 9 ; 2 ) severs the MSE gradient pathway entirely via \mathrm{sg}(\cdot) , preventing continuous gradients from reaching the VLM backbone. This forces the system to rely on discrete proxy tokens (e.g., FAST ( 23 ) ) as the sole VLM learning signal, discarding the rich continuous supervision available in demonstration data. While stop-gradient preserves VQA (our stop-gradient baseline shows VQA loss remaining roughly constant), this preservation comes at the cost of encoding robot actions as discrete tokens in the LLM vocabulary—a representational proxy that compresses continuous 7 -DoF motor commands into a foreign code-space. 2. Low-rank adaptation (LoRA) ( 12 ; 14 ) restricts the capacity of learning by confining weight updates to a low-rank subspace ( r{=}16 , \alpha{=}32 on LLM layers). However, LoRA does not restrict the geometry of learning: if the MSE gradient projects destructively onto the VQA manifold within the low-rank subspace, LoRA will faithfully execute that destructive update. Our experiments confirm this: our LoRA baseline slows degradation but still causes a steady increase in VQA loss ( Figure 3 ), demonstrating that low-rank constraints alone are insufficient against cross-modal gradient asymmetry. Critically, both strategies in production systems additionally rely on mixed-batch VQA co-training —interleaving VQA data (up to 50\% ) into the robotic training batches to artificially maintain the pre-trained distribution ( 9 ; 12 ) . This “co-training tax” doubles the effective compute cost and requires maintaining access to the original VQA corpus during robotic fine-tuning. AEGIS: gradient surgery on the Wasserstein manifold. We propose AEGIS ( A nchor- E nforced G radient I solation S ystem), a framework that resolves the cross-modal gradient asymmetry at its geometric root. AEGIS consists of three components: 1. Static Wasserstein anchor ( Section 4.1 ): Before fine-tuning, AEGIS pre-computes per-layer Gaussian statistics (\bm{\mu}^{0}_{\ell},\bm{\sigma}^{0\,2}_{\ell}) from masked VQA forward passes across all 26 transformer layers of the VLM. These statistics define a reference point on the Wasserstein- 2 manifold of activation distributions. The anchor deliberately uses the full attention mask (images + text) to capture the complete semantic manifold, while excluding padding tokens to prevent phantom attractors at non-semantic zero-vectors. 2. Wasserstein- 2 transport penalty ( Section 4.2 ): At each training step, AEGIS computes the squared \mathcal{W}_{2} distance between the current and anchor activation statistics, decomposed via the Bures metric into mean-shift and standard-deviation-mismatch terms. This penalty generates a second set of gradients through the same computation graph. 3. Layer-wise orthogonal gradient projection ( Section 4.3 ): A sequential dual-backward decomposes the task (FM) and anchor (OT) gradients. For each transformer layer, AEGIS computes a single global projection coefficient \alpha_{\ell}=\langle{\bm{g}}_{\text{task}}^{\ell},{\bm{g}}_{\text{ot}}^{\ell}\rangle/\|{\bm{g}}_{\text{ot}}^{\ell}\|^{2} and, when destructive ( \alpha_{\ell}<0 ), subtracts the interfering component: {\bm{g}}_{\text{final}}^{\ell}={\bm{g}}_{\text{task}}^{\ell}-\alpha_{\ell}\,{\bm{g}}_{\text{ot}}^{\ell} . This Gram–Schmidt step bends the task gradient orthogonal to the anchor manifold’s normal direction, preserving all constructive gradient energy while eliminating the destructive projection. The layer-wise granularity—neither global (which suffers from a “zero-sum loophole” where opposing layers cancel) nor per-tensor (which destroys attention-head synchronisation)—is the geometric sweet spot that respects the internal structure of transformer blocks. AEGIS is purely a backward-pass intervention: the forward pass, loss functions, and model architecture remain entirely unchanged . The VLM sees only action data during training—no VQA co-training, no replay buffer, no discrete proxy tokens. Contributions. Our contributions are: 1. We identify cross-modal gradient asymmetry as the root cause of VQA degradation during VLA fine-tuning, demonstrating how highly concentrated MSE gradients overwrite delicate semantic parameters ( Section 3 ). 2. We formalise AEGIS , a buffer-free framework that constructs a Wasserstein- 2 Gaussian anchor over the pre-trained activation manifold and performs per-layer Gram–Schmidt orthogonal projection to eliminate destructive gradient interference while preserving constructive task learning ( Section 4 ). 2 Related Work Vision-Language-Action models. RT-2 ( 3 ) demonstrated that VLMs can directly output tokenised actions. \pi_{0} ( 2 ) integrates a flow-matching action expert with a VLM backbone via cross-attention, establishing the hybrid architecture that our work builds upon. OpenVLA ( 16 ) open-sourced a 7B VLA with discrete action tokenisation. KIVA ( 9 ) introduced the stop-gradient operator to prevent flow-matching gradients from corrupting VLM weights, enabling use of the FAST tokeniser ( 23 ) for discrete action representation. Concurrently, VLM2VLA ( 12 ) identifies action gradients as destructive and proposes LoRA fine-tuning on the VLM’s LLM layers. Our LoRA baseline directly replicates the VLM2VLA configuration and shows that it fails to prevent VQA erosion, as destructive MSE gradient components persist and overwrite the VQA manifold even within the low-rank subspace. AEGIS is complementary to both: it operates on

Read the complete text