ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling

Konstantinos Georgiou, Maofeng Tang, Hairong Qi · 2026

Read ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling by Konstantinos Georgiou, Maofeng Tang, Hairong Qi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-31201.txt.

Introduction

Georgiou and two colleagues present ExPLoRe, a method for masked image modeling with several learning goals at once. Masked image modeling hides parts of an image and trains a model to recover signals such as token distillation, class alignment, and pixel reconstruction. Older work weights those goals with one global number, ignoring that patches differ.

ExPLoRe reuses Soft Mixture of Experts dispatch weights as per-patch loss weights, so each patch stresses goals differently. Letting loss gradients flow into the router is the core step; blocking them drops results by 1.6 percent. On ImageNet-1K the method lifts probe scores, reaches 85.3 percent fine-tuning accuracy, and with new adaptation recipes closes the segmentation gap on ADE20K.

Catalog record

CREATORKonstantinos Georgiou, Maofeng Tang, Hairong Qi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.31201

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ExPLoRe: Expert Patch-Level Loss Routing for Multi-Objective Masked Image Modeling Konstantinos Georgiou, Maofeng Tang, Hairong Qi Abstract Multi-objective masked image modeling (MIM) combines complementary learning signals (token distillation, CLS alignment, and pixel reconstruction) but existing methods weight these objectives with global scalars, ignoring spatial heterogeneity across patches. We present ExPLoRe (Expert Patch-Level Loss Routing), which repurposes Soft Mixture of Experts (MoE) dispatch weights as learned, per-patch loss coefficients. The key mechanism is loss-coupling : allowing loss gradients to flow through dispatch weights to the router enables content-dependent specialization, where different patches receive different emphases across objectives. A detach ablation confirms loss-coupling as the core mechanism, degrading performance by 1.6% when gradients are blocked. On ImageNet-1K with ViT-Base, ExPLoRe improves over non-MoE baselines on two objective combinations (Token+CLS: +0.5% k-NN, +4.4% linear probe; Token+Pixel: +2.2% k-NN), achieving 80.6% linear probe and 85.3% finetuning accuracy, competitive with published methods. For downstream transfer, we develop adaptation recipes (Freeze Routing, Expert Dropout, and Freeze Attention) that improve MoE finetuning by +1.5% over the vanilla MoE, and close a 2.5–2.9 mIoU segmentation gap so that MoE models match or exceed non-MoE baselines on ADE20K. 1 Introduction Figure 1 : Expert dispatch-weight visualization. Per-patch dispatch weights from a trained 2-expert ExPLoRe model overlaid on input images (warm = high weight, cool = low weight). The two experts learn complementary spatial specialization without explicit supervision: one expert assigns higher loss emphasis to foreground regions while the other focuses on background and context. Masked image modeling (MIM), inspired by BERT’s masked language modeling [ 10 ] , has emerged as a powerful self-supervised learning paradigm for vision [ 3 , 15 ] . Teacher-guided approaches [ 21 , 16 , 12 ] leverage frozen feature extractors such as CLIP [ 25 ] to provide semantic targets for student encoders. Recent works combine multiple learning signals including token-level distillation, CLS alignment, and pixel reconstruction; but how to weight these diverse objectives optimally remains an open question that is largely independent of the specific teacher used. The Patch-Level Heterogeneity Problem. Existing multi-task learning methods [ 8 , 19 ] balance losses at the global level, applying a single scalar weight per objective uniformly across all spatial locations. However, different image regions benefit from different learning signals: semantically rich patches are better served by teacher distillation, while texture-heavy background regions benefit from pixel reconstruction. Global methods cannot capture this spatial variation. As we demonstrate, gradient-based balancing (GradNorm [ 8 ] ) can even fail catastrophically when objectives have spatially heterogeneous importance. To our knowledge, no prior work addresses patch-level loss weighting for masked image modeling. We present ExPLoRe 1 1 1 Code available at https://github.com/aicip/ExPLoRe . ( Ex pert P atch- L evel L o ss R outing), which addresses patch-level heterogeneity by repurposing Soft Mixture of Experts (Soft-MoE) [ 24 ] dispatch weights as learned, per-patch loss coefficients. Unlike traditional MoE methods that use routing for capacity scaling [ 29 , 13 , 26 ] or task-specific expert assignment [ 5 , 34 ] , ExPLoRe uses continuous, sample-adaptive weighting through a loss-coupling mechanism: gradients from each loss objective flow back through the dispatch weights to the router, enabling the model to learn content-dependent specialization. We evaluate two objective combinations ( Token+CLS and Token+Pixel ) with expert scaling from 2 to 64 under sparse encoding [ 15 ] , and develop downstream transfer recipes for competitive finetuning. Figure 1 visualizes the resulting routing patterns. ExPLoRe provides content-dependent per-patch loss emphasis that prior methods lack; competitive results across classification and dense prediction demonstrate this mechanism does not sacrifice representation quality. Our contributions center on a single mechanism with two supporting analyses: • Core contribution: dispatch-weight loss weighting. We introduce a mechanism that repurposes Soft-MoE dispatch weights as per-patch loss coefficients for multi-objective MIM. Loss-coupling, where loss gradients flow through dispatch weights to update the router, is the core innovation enabling learned per-patch specialization. We demonstrate consistent improvements over non-MoE baselines across two objective combinations, achieving 80.6% linear probe accuracy competitive with published methods (Table 6 ). • Routing dynamics analysis. We empirically characterize routing behavior: entropy regularization governs a sharp transition between stable routing and catastrophic collapse; optimal regularization is expert-count dependent; and a detach ablation validates loss-coupling as the mechanism driving specialization (Sections 4.2 and 4.2 ). • Downstream transfer recipes. We develop complementary adaptation strategies, Freeze Routing (FR), Expert Dropout (ExD), and Freeze Attention (FA), that compose naturally. The combined FR+FA+ExD recipe exceeds the non-MoE baseline on both classification (+0.5%) and semantic segmentation, closing a 2.5–2.9 mIoU gap (Section 4.4 ). 2 Related Work Masked Image Modeling. MIM extends masked language modeling (MLM) from NLP to vision. BEiT [ 3 ] uses dense encoders processing all patches with mask tokens, while MAE [ 15 ] uses sparse encoders processing only visible patches with shuffling for efficiency. SimMIM [ 33 ] uses simple linear projections, iBOT [ 38 ] combines masking with self-distillation, and MaskFeat [ 30 ] uses HOG features as targets. Self-distillation approaches include data2vec [ 2 , 1 ] (EMA representations), CAE/CAEv2 [ 6 , 36 ] (context autoencoding), SdAE [ 7 ] (pixel + self-distillation), and BootMAE [ 11 ] (bootstrapping). Teacher-Guided MIM with CLIP. CLIP [ 25 ] enables MIM methods with semantic targets. MaskDistill [ 21 ] provides a unified view of masked image modeling, formulating MIM as \mathcal{L}(\mathcal{N}(\mathcal{T}(I)),\mathcal{H}(\mathcal{S}(I_{m}))) with teacher \mathcal{T} , student \mathcal{S} , projector heads \mathcal{N} / \mathcal{H} , and shows CLIP features outperform pixels or discrete tokens as reconstruction targets. Our base architecture builds upon MaskDistill’s framework. BEiT v2 [ 22 ] combines tokenizers with CLIP distillation. MILAN [ 16 ] uses CLIP attention for semantic masking and combines CLS with patch distillation. MaskCLIP [ 12 ] explores masked self-distillation within CLIP. Multi-Task Learning and Loss Balancing. Multiple methods balance competing objectives: uncertainty weighting [ 18 ] models task-dependent uncertainty, GradNorm [ 8 ] normalizes gradient magnitudes, PCGrad [ 35 ] projects conflicting gradients, MGDA [ 28 ] seeks Pareto-optimal solutions, and random weighting [ 19 ] samples task weights stochastically. Crucially, all operate at the global level, applying a single scalar weight per objective uniformly across all spatial locations. This ignores the spatial structure of images, where different regions may benefit from different loss emphasis. To our knowledge, no prior work addresses per-patch adaptive loss weighting for masked image modeling, which we identify as a key gap. Mixture of Experts. Sparse MoE methods [ 29 , 13 ] route tokens to expert subsets for conditional computation but introduce load balancing challenges. Soft-MoE [ 24 ] addresses this through fully-differentiable soft routing, where dispatch and combine weights computed via softmax avoid discrete decisions and their associated training difficulties. V-MoE [ 26 ] applied sparse MoE to vision transformers; subsequent work [ 20 , 14 ] compared routing strategies, and multi-task applications [ 5 , 34 ] demonstrated MoE effectiveness in vision. CR-MoE [ 17 ] first applied MoE to self-supervised learning (contrastive), discovering routing consistency challenges. However, applications to masked image modeling with dynamic task weighting have not been investigated. MoE downstream transfer remains understudied; ViMoE [ 14 ] finds that only 2–5 MoE layers suffice but focuses on supervised

Read the complete text