Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models
Yishun Lu and Wes Armour · 2026
Read Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models by Yishun Lu and Wes Armour (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-16165.txt.
Introduction
Lu and Armour tackle modality clashes in autoregressive multimodal training, where image making and text understanding fight and destabilize learning. Simple optimizers like AdamW suffer when gradients differ across modalities. Their ML-FOP-SOAP framework adds multi-level variance correction, with a Fisher-orthogonal projection that damps variance-driven clashes. On Janus and Emu3 it improved sample efficiency by up to 1.4-fold and wall-clock speed by up to 1.5-fold over AdamW, with stable training at batch size 8,192.
Catalog record
| CREATOR | Yishun Lu and Wes Armour |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54378 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.16165
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models Yishun Lu and Wes Armour Abstract Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality competition that destabilizes optimization and limits large-batch scaling. We show that first-order optimizers such as AdamW are vulnerable to cross-modality gradient heterogeneity, while second-order preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment. Building on this insight, we propose ML-FOP-SOAP , a second-order optimization framework with Multi-Level Variance Correction. Our Fisher-Orthogonal Projection suppresses variance-induced modality conflicts, reducing the trade-off between visual generation and textual understanding. To make this practical under large gradient accumulation, we introduce a hierarchical folding strategy that captures fine-grained variance with low micro-step overhead. Experiments on Janus and Emu3 show consistent gains across both modalities and stable training at batch size 8192. Compared with AdamW, our method improves sample efficiency by up to 1.4 × 1.4\times and accelerates wall-clock training by up to 1.5 × 1.5\times , offering a robust optimizer for scaling multimodal foundation models. 1 Introduction In recent years, unifying image generation and text understanding under the autoregressive next-token prediction paradigm has become a major trend in the development of vision-language foundation models. However, this deep multi-modal integration introduces a severe modality competition problem during pretraining, which often makes joint training highly unstable and convergence extremely slow. To address this challenge, most existing approaches in the literature primarily focus on model architecture. For example, researchers have attempted to physically disentangle modality-specific representations by designing decoupled visual encoders, introducing multi-way attention mechanisms, or constructing dedicated alignment modules [ 21 , 23 , 25 , 9 , 2 ] . Beyond architectural modifications, existing efforts to alleviate modality conflict, such as dynamic loss weighting, curriculum scheduling, and gradient balancing [ 3 , 4 , 7 , 19 ] , remain largely external to the optimizer. In practice, parameter updates are still governed by first-order methods like AdamW [ 13 ] , which rely on diagonal moment estimates. These methods fail to model the underlying local curvature of the loss surface or the geometric coupling between modalities. This reliance on first-order optimization becomes a critical bottleneck because multi-modal training induces gradient estimators with vastly different noise patterns and curvature sensitivities [ 8 ] . Specifically, visual objectives often involve high-dimensional and spatially diffuse prediction errors, whereas textual objectives are typically concentrated on a more compact semantic manifold [ 26 ] . This geometric mismatch creates a complex cross-modal covariance structure. Consequently, under standard diagonal preconditioning, modalities associated with larger gradient variance (vision) exert a disproportionate influence on the update trajectory, suppressing lower-variance but semantically dense signals (text) [ 18 , 24 ] . To resolve this variance conflict at the optimization level, the recently proposed Fisher-Orthogonal Projection (FOP) [ 14 ] accelerates convergence by projecting gradients in Fisher geometry. However, its original reliance on K-FAC [ 15 ] is memory-prohibitive for the massive vocabularies and wide hidden layers of modern foundation models. To address this, we leverage structured two-dimensional (2D) tensor preconditioners. Interestingly, our empirical analysis reveals that while standard methods like Shampoo [ 20 ] are easily destabilized by early-stage visual outliers, the SOAP optimizer [ 22 , 1 ] provides a highly robust, memory-efficient surrogate for local optimization metrics. Yet, even with SOAP, we observe a lingering trade-off where textual understanding is inadvertently sacrificed to accommodate high-entropy visual noise. Thus, integrating FOP with SOAP becomes essential to fully neutralize these cross-modality conflicts. Furthermore, scaling these unified models to ultra-large global batch sizes (e.g., 8192) requires massive gradient accumulation across numerous micro-steps. Applying variance correction only at the coarse boundaries of a long accumulation window fails to preserve the fine-grained gradient discrepancies arising within the window. Inspired by the telescoping coarse-to-fine viewpoint of the Multi-Level Monte Carlo (MLMC) paradigm [ 5 ] , we propose a novel Multi-level Hierarchical Gradient Folding mechanism via Fisher-Orthogonal Projection (ML-FOP) . Instead of classical scalar expectation estimation, our approach recursively fuses computationally lightweight cumulative snapshots to preserve intra-step multi-scale variance to achieve multi-level variance correction. Through this methodology, we present a new optimization framework that natively mitigates modality competition and enables the training of unified autoregressive models with ultra-large global data batches, dramatically improving hardware utilization on modern GPU clusters. Specifically, our contributions are as follows: • Second-Order Optimization for Modality Competition. We systematically investigate the optimization dynamics of unified multi-modal autoregressive models. We reveal that while first-order methods (AdamW [ 13 ] ) and standard uncentered second-order methods (Shampoo [ 20 ] ) struggle with extreme visual gradient outliers, the SOAP optimizer [ 22 ] inherently mitigates this cross-modality heterogeneity, establishing a robust foundation for multi-modal alignment. Compared to AdamW, our ML-FOP-SOAP framework delivers a 1.4\times gain in sample efficiency and a 1.5\times acceleration in end-to-end wall-clock time, uniquely preserving robust performance across both modalities. • Variance Control via Fisher-Orthogonal Projection (FOP). Building upon the SOAP baseline, we introduce a geometry-aware FOP mechanism. We demonstrate that projecting gradient differences orthogonally to the primary descent trajectory actively neutralizes modality competition. This effectively eliminates the conventional trade-off between textual understanding and visual generation, yielding strict Pareto improvements across both tasks. • Efficient Multi-Level FOP for Massive Accumulation. To make FOP tractable under massive gradient accumulation, we propose an efficient Multi-Level hierarchical folding mechanism (ML-FOP). This dyadic telescoping strategy captures fine-grained, cross-modality variance across accumulation segments without the prohibitive computational wall-clock overhead of applying projections at every micro-step. • Ultra-Large-Batch multi-modal Alignment Benchmark. We validate our framework on representative unified architectures (Janus and Emu3) scaled on massive datasets (LLaVA-3M and 12M [ 12 , 11 ] ). We successfully scale the global batch size up to 8192, a regime where existing optimizers suffer catastrophic degradation. 2 Related Work 2.1 Native Unified Vision-Language Models The dominant paradigm in vision-language foundation models has recently shifted towards unifying image and text modalities within a single autoregressive next-token prediction framework [ 21 , 25 , 23 ] . Unlike earlier compositional architectures that connect a pre-trained vision encoder (e.g., CLIP) to a Large Language Model via projection layers [ 9 , 2 ] , native autoregressive models map all continuous visual signals into discrete tokens, treating both modalities equally. While this architectural convergence enables seamless generation and understanding, it inevitably exacerbates modality competition during joint pre-training. Our work does not alter these advanced model architectures; rather, we provide the optimization infrastructure necessary to stabilize their training and scale them to unprecedented batch sizes. 2.2 Modality Competition and Multitask Optimization The phenomenon of modality conflict is closely related to gradient interference in multitask learning. Traditional approaches to alleviate this include dynamic loss weighting [ 3 ] , gradient projection techniques, and curriculum-based data scheduling [ 4 , 7 , 19 ] . However, these methods operate largely external to the optimizer itself. Parameter updates in foundation models are still universally governed by first-order optimizers like AdamW [ 13 ] . AdamW relies on diagonal preconditioning, which independently scales coordinates but fundamentally fails to capture the complex, non-diagonal cross-modal covariance structure and the distinct local geometry induced by high-variance visual objectives and dense textual signals [ 8 ] . 2.3 Efficient Second-Order