Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs

Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang · 2026

Read Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs by Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03879.txt.

Introduction

Ding and co-authors measure what each vision encoder really adds inside multi-encoder language models. Fusing varied visual streams is a growing design trend. How encoders interact under joint training was poorly understood. Rankings from masking a fixed checkpoint may mislead. The team retrains all 31 non-empty subsets of five common encoders on the Cambrian-1 suite. The effort totals about 20,000 GPU hours.

Retraining reshuffles encoder rankings, including which encoder leads overall. Each encoder's role splits into Capacity and Necessity. Capacity is its solo score; Necessity is the drop when it leaves the full pool. The two axes do not swap for each other. The best pair joins a high-Capacity anchor with an adaptive complement, matching all five encoders. Extra encoders add little beyond that pair. Pre-projector rank explains the leftover score spread at fixed size. Designers get concrete primitives for leaner multi-encoder builds.

Catalog record

CREATORWei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE45 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03879

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs Wei Ding, Yudong Zhang, Ruobing Xie, Xingwu Sun, Jiansheng Chen, Yu Wang Abstract As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large vision-language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline ( ≈ \approx 20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder’s contribution into two axes, Capacity , the score an encoder reaches on its own, and Necessity , the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank expands under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder–projector interface. Together, the Capacity–Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it. 1 Introduction Large Vision-Language Models (LVLMs) ( 21 ; 38 ) have made substantial progress on multimodal tasks, and combining multiple heterogeneous vision encoders has been proposed as one route to further gains. Eagle ( 30 ) is a representative example, fusing features from ConvNeXt ( 23 ) , EVA-02 ( 5 ) , CLIP ( 27 ) , Pix2Struct ( 18 ) , and SAM ( 15 ) through channel concatenation, a recipe also adopted by Prismatic VLMs ( 13 ) and subsequent channel-concat designs ( 10 ) ; the channel-concat family is one of three fusion mechanisms surveyed in Section 2 , alongside token-sequence concatenation and cross-attention aggregation (which Cambrian-1 ( 32 ) and BRAVE ( 12 ) adopt). The same trend extends beyond benchmark VLMs to vision-language-action policies ( 14 ) , and foundation models for world modeling and physical AI ( 26 ) continue to fuse heterogeneous perception streams, making principled methodology for evaluating multi-encoder fusion broadly relevant. Vision encoders are much smaller than the language backbone, so adding them is an attractive design lever if the gains are real. Some studies have attempted to conduct ablation analyses on the roles of different encoders in multi-encoder LVLMs, for example, 35 study this question by masking individual encoder branches at inference time and measuring the resulting performance drops, formalizing the protocol as the Conditional Utilization Rate (CUR). Two limitations remain: this inference-time evaluation departs from the actual training pipeline and tends to produce unreliable conclusions, and no prior method evaluates the full set of encoder subsets. We adopt the five-encoder family as a concrete instance to study. Figure 1 : Paradigm preview. (A) IM and TR rank a different encoder first: EVA-02 under IM, ConvNeXt under TR. The two protocols also swap at rank 2, while ranks 3 to 5 agree. (B) Best-at- k overall score. With CLIP alone as the baseline and the full pool as the ceiling, ConvNeXt alone closes 85% of the gap and CLIP+ConvNeXt closes 97%. The third and fourth encoders add little. (C) Capacity–Necessity plane. The five encoders fall into the four regions of the plane. The arrow points to the best two-encoder pair, ConvNeXt+CLIP, which is not the pair formed by the two highest-Capacity encoders. We address these limitations by retraining all 31 encoder subsets under a unified training and evaluation pipeline. Taking the Inference-time Masking (IM) results of 35 as the reference and our Training-time Removal (TR) results as the ground truth, we identify two systematic discrepancies. IM and TR rank a different encoder first, and they disagree on which two-encoder combination is best, in a way that changes the practical recommendation. IM also shows higher cross-subset variance than TR for most encoders, indicating that retraining each subset gives a more stable measurement of individual encoder contributions. Building on this protocol, three findings shape our recommendation for compact-pool design. First, adding vision encoders yields strongly diminishing returns: a well-chosen two-encoder model recovers nearly all of the full-pool score, while the third and fourth encoders contribute little, and only the fifth recovers a small Vision-Centric margin. Second, encoder contributions decompose along two axes we call Capacity and Necessity. A single attribution score conflates how well an encoder performs alone with how much of its contribution is irreplaceable by the others, and disentangling them yields a five-class encoder taxonomy together with a non-obvious pair-selection rule: pairing the two highest-Capacity encoders is suboptimal. Third, per-encoder pre-projector effective rank explains residual score variation at fixed parameter count, connecting compact-pool selection to a representation-level signal whose change under joint training predicts pair quality. Figure 1 previews the protocol contrast and the two-axis decomposition. The contributions of this paper are summarized as follows: (1) Attribution. A retrain-and-remove protocol that yields more stable, decision-relevant rankings than inference-time masking, and disagrees with masking on which encoder ranks first. (2) Decomposition. The Capacity and Necessity axes, which expose substitutability that single-score attribution conflates and yield a Universal Core with Adaptive Complement pair-selection pattern that the two-highest-Capacity heuristic misses. (3) Mechanism. Per-encoder pre-projector effective rank as a representation-level selection signal independent of parameter count, supporting an anchor with rank-expanding complement principle for compact-pool design. 2 Related Work Multi-Encoder LVLMs. Multi-encoder LVLMs combine heterogeneous visual streams via three fusion mechanisms ( 30 ) : token-sequence concatenation ( 20 ; 4 ) , cross-attention aggregation ( 19 ; 32 ; 12 ; 25 ) , and channel-level concatenation ( 13 ; 10 ) ; 32 provide the largest per-encoder benchmarking to date. A routing line activates encoders conditionally per input ( 39 ; 17 ; 36 ) , with 36 showing that one routed encoder can match four fused ones. A complementary direction compresses a multi-encoder pool into one backbone via distillation ( 28 ; 8 ; 2 ; 34 ) , implicitly assuming every teacher’s contribution is worth preserving. We adopt Eagle-X5 ( 30 ) as our testbed (channel-concat keeps token count independent of the active encoder set, making 31-subset enumeration tractable). Relative to all three lines above, our contribution is methodological rather than architectural : the 31-subset audit quantifies how much of each teacher’s contribution actually survives joint training, providing direct guidance for distillation pipelines. Encoder Attribution and Representation Analysis. Identifying which encoders a multi-encoder LVLM relies on after joint training is a measurement problem. The closest precursor, 35 , formalises this for Eagle-X5 via Conditional Utilization Rate (CUR) and Information Gap (IG), computed by Inference-time Masking (IM) on a fixed checkpoint; earlier inference-time subset removal appears

Read the complete text