AECF: Robust Multimodal Learning via Entropy-Gated Contrastive Fusion

Leon Chlon, Maggie Chlon, MarcAntonio M. Awada · 2025

Read AECF: Robust Multimodal Learning via Entropy-Gated Contrastive Fusion by Leon Chlon, Maggie Chlon, MarcAntonio M. Awada (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15417.txt.

Introduction

Leon Chlon, Maggie Chlon, and MarcAntonio Awada present a fusion layer that keeps multimodal models steady when inputs go missing, in a 2025 paper. Real systems lose channels: a robot's microphone dies, or a clinic record lacks a lab test. Standard fusion layers force a trade between robustness and honest confidence scores. Their layer, AECF, adjusts its entropy setting for each case and keeps confidence scores ordered across all subsets of inputs.

A masking schedule during training grows out of the model's own uncertainty. On two benchmarks, the layer raised accuracy on half-missing inputs by 18 points while cutting calibration error sharply. It added only about one percent to run time, and the main networks stayed frozen. The authors offer it as a drop-in fix for deployed systems.

Catalog record

CREATORLeon Chlon, Maggie Chlon, MarcAntonio M. Awada
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15417

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AECF: Robust Multimodal Learning via Entropy-Gated Contrastive Fusion Leon Chlon, Maggie Chlon, MarcAntonio M. Awada Abstract Real-world multimodal systems routinely face missing-input scenarios—for example a robot loses audio in a factory or a clinical record omits lab tests at inference time. Standard fusion layers either preserve robustness or calibration but never both. We introduce Adaptive Entropy-Gated Contrastive Fusion (AECF), a single light-weight layer that (i) adapts its entropy coefficient per instance, (ii) enforces monotone calibration across all modality subsets, and (iii) drives a curriculum mask directly from training-time entropy. On AV-MNIST and MS-COCO, AECF improves masked-input mAP by +18 pp at a 50 % drop rate while reducing ECE by up to 2 × \times , yet adds < < 1 % run-time. All back-bones remain frozen, making AECF an easy drop-in layer for robust, calibrated multimodal inference. 1 Introduction Multimodal models now underpin image captioning, audio–vision retrieval and embodied perception systems that must operate under real-world noise, occlusion and partial sensor failure. In practice, missing modalities are the rule, not the exception: a robot may lose audio in a crowded factory, or a medical record may omit lab tests at test time. Unfortunately, standard mixture-of-experts (MoE) fusion layers either collapse to the dominant modality, harming robustness ( Neverova et al., 2015 ) , or they learn to hedge by emitting over-confident probabilities ( Tang et al., 2024 ) . Reliable deployment therefore requires two simultaneous properties : 1. Robust accuracy when any subset of modalities is present, and 2. Well-calibrated confidences that remain monotone as information is added or removed. Existing approaches address at most one of these axes. Tang et al. (2024) graft a fixed- \lambda entropy penalty onto a soft gate to limit collapse but do not calibrate; Ma et al. (2023) calibrate two experts via a contrastive loss yet have no notion of curriculum masking; ModDrop ( Neverova et al., 2015 ) randomises modality dropout during training but cannot adapt to sample-wise uncertainty. Recent calibration unexplored and incurring significant compute. Image Enc. f_{1} Text Enc. f_{2} Audio Enc. f_{3} Concat+Norm g_{\phi} Weighted Fusion Uncertainty u_{m} g_{\alpha} Task Head h_{\psi} Curriculum Mask \pi_{t}(S) gate entropy Figure 1 : AECF pipeline. A two–layer gate mixes frozen encoder features. During training the gate entropy H(p) is penalised (coefficient \lambda_{t} follows the epoch-dependent schedule in §3.3) while inputs are randomly masked with probability \pi_{t} that linearly ramps to \pi_{\max} . Both schedules are deterministic once training starts; at test time no masking is applied. This paper introduces Adaptive Entropy-Gated Contrastive Fusion AECF , a light-weight fusion layer that achieves both axes without touching frozen back-bones. AECF consists of three tightly coupled modules (Fig. Fig. 1 ): 1. Meta-adaptive entropy gate. A per-instance coefficient \lambda(x) modulates the gate’s entropy, raising regularisation when predictive uncertainty is high (§ 3 ). Unlike the global penalty of Tang et al. (2024) , the adaptive \lambda(x) yields a formal bound on worst-subset regret (Lemma 4 ). 2. Contrastive Expert calibration (CEC). A novel contrastive loss enforces monotone confidences across all 2^{M}\!-\!1 modality subsets, extending the two-expert calibration of Ma et al. (2023) . Proposition 2 proves CEC decreases positive-ECE on the entire subset lattice. 3. Adaptive Curriculum Masking (ACM). Training masks are sampled by a teacher that maximises the current gate entropy , producing adversarial missing patterns that sharpen the mixture. This differs from ModDrop’s random masking and avoids the 2^{M} enumeration cost via an O(M) algorithm. Contributions. • The first single fusion layer that is simultaneously instance-adaptive (per-sample \lambda ), lattice-calibrated (CEC), and curriculum-aware (entropy-driven masking). Section 5 shows that stitching any two of these ideas is still inferior by \geq 6 pp mAP or 1.5 \times ECE. • Formal guarantees: a worst-case subset regret bound for the adaptive entropy gate and a PAC bound for CEC that ensures ECE cannot increase as modalities are added. • Extensive evaluation on AV-MNIST and MS-COCO: AECF improves masked-input mAP by up to +18 pp while halving ECE, with < 1 % compute overhead. 2 Related Work Foundations: probabilistic mixtures and classical gating. Mixture–of–Experts (MoE) with probabilistic gating originated in the neural–network literature of the early 1990s. Robert A. Jacobs and Hinton (1991) and Jordan and Jacobs (1994) trained hierarchical MoEs by expectation–maximisation, letting subnetworks specialise on distinct input regions. Contemporary “committee machine” work likewise combined modality-specific experts under a learned gate. These ideas anticipated modern deep MoEs by formalising the trade-off between expert specialisation and mixture uncertainty. Deep MoE and multimodal robustness. Sparsely gated MoEs resurfaced with the Mixture-of-Experts Layer of Shazeer et al. (2017) and the Switch Transformer ( Fedus et al., 2021 ) , where a lightweight router activates the top- k experts per input. Extensions to vision include Selective-Kernel networks and FiLM-style channel gates, though not strictly MoE. Robust multimodal fusion often relies on modality dropout : Neverova et al. (2015) randomly drop entire channels, forcing the network to exploit cross-modal correlations. Later variants learn which modalities to drop via a trainable mask ( Alfasly et al., 2022 ) , or reconstruct masked modalities through masked-projection objectives ( Nezakati and Xie, 2024 ) . Adapter modules can also compensate for missing inputs: Reza and Timofte (2023) insert parameter-efficient adapters that modulate hidden activations so performance degrades gracefully. Recent MoE fusion architectures push dynamic gating into the multimodal setting: Cao et al. (2023) gate local-detail and global-context experts for infrared–visible blending, while Han et al. (2024) propose a transformer whose gate allocates experts across an arbitrary number of modalities. All these methods aim to prevent modality collapse —over-reliance on a single dominant modality—yet their gating coefficients are fixed and their masking schedules heuristic. Calibration and uncertainty under missing modalities. Deep classifiers are notoriously mis-calibrated ( Guo et al., 2017 ) , and the problem is amplified in multimodal models: Ma et al. (2023) show confidence often increases when a modality is removed. They regularise logits so that confidence is non-increasing in missingness, but only compare full versus single-drop cases. Temperature scaling and focal-style objectives ( Mukhoti et al., 2024 ) reduce ECE on the full input, yet do not enforce consistency across subsets. Bayesian approaches such as MC Dropout ( Gal and Ghahramani, 2016 ) and deep ensembles ( Lakshminarayanan et al., 2017 ) provide uncertainty estimates but are computationally heavy at test time. Recent multimodal-specific schemes weight modalities by per-modality uncertainty ( Tang et al., 2024 ) , again focussing on binary present/absent settings rather than the full powerset. Positioning of AECF. AECF unifies these three strands. Its meta-adaptive entropy-regularised gate generalises classical log-barrier MoE theory to an instance-wise uncertainty coefficient, provably shrinking worst-subset regret. Adaptive curriculum masking replaces fixed or random dropout with a feedback-driven schedule that adversarially targets dominant modalities. Finally, contrastive expert calibration enforces ranking consistency across all 2^{M}-1 subsets, yielding monotone ECE improvement—a guarantee absent from prior calibration or uncertainty techniques. ( Wang et al., 2025 ) in that both keep encoder gradients and rely on heavy pre-training; AECF freezes encoders and focuses on the fusion layer alone, making it complementary. 3 AECF This section formalises the problem, details the three AECF modules, and presents an end-to-end training algorithm. 3.1 Problem setting and notation Let \mathcal{X}=\{x^{(1)},\ldots,x^{(M)}\} be M modalities

Read the complete text