ConsensusDrop: Fusing Visual and Cross-Modal Saliency for Efficient Vision Language Models Dhruv Parikh, Haoyang Fan, Rajgopal Kannan, Viktor Prasanna Abstract Vision–Language Models (VLMs) are expensive because the LLM processes hundreds of largely redundant visual tokens. Existing token reduction methods typically exploit either vision-encoder saliency (broad but query-agnostic) or LLM cross-attention (query-aware but sparse and costly). We show that neither signal alone is sufficient: fusing them consistently improves performance compared to unimodal visual token selection (ranking). However, making such fusion practical is non-trivial: cross-modal saliency is usually only available inside the LLM (too late for efficient pre-LLM pruning), and the two signals are inherently asymmetric, so naive fusion underutilizes their complementary strengths. We propose ConsensusDrop , a training-free framework that derives a consensus ranking by reconciling vision encoder saliency with query-aware cross-attention, retaining the most informative tokens while compressing the remainder via encoder-guided token merging. Across LLaVA-1.5/NeXT, Video-LLaVA, and other open-source VLMs, ConsensusDrop consistently outperforms prior pruning methods under identical token budgets and delivers a stronger accuracy–efficiency Pareto frontier—preserving near-baseline accuracy even at aggressive token reductions while reducing TTFT and KV cache footprint. Our code will be open-sourced. 1 Introduction The recent advancement in Large Language Models (LLMs), encompassing both closed- 57 ; 3 and open-source paradigms 56 ; 58 ; 1 ; 92 , has fueled a new generation of Vision-Language Models (VLMs). These models achieve complex multimodal reasoning—either natively 55 ; 57 or through architectural extensions 46 ; 2 —and have extended capabilities to specialized domains such as document understanding, chart interpretation, and long-video comprehension 15 ; 63 ; 33 ; 34 ; 61 ; 85 ; 13 . Standard VLMs typically employ a pre-trained vision encoder 64 ; 87 to map images into patch tokens, followed by a multimodal projector 46 ; 45 ; 42 that aligns these embeddings with the LLM. However, this paradigm incurs significant computational overhead. A single image is typically encoded into hundreds or thousands of tokens (e.g., 576 in LLaVA-1.5 46 , 2880 in LLaVA-Next 45 ), vastly exceeding the textual input containing fewer than a hundred tokens. This redundancy leads to prohibitive prefill latency and excessive KV-cache consumption during decoding, hindering scalable deployment in latency-sensitive applications such as robotics and autonomous driving 30 ; 62 ; 78 . Figure 1 : Motivation: Complementary failure modes of visual token sparsification. Given a single image and two questions requiring distinct visual evidence (global structure vs. localized fine-grained text), different token selection criteria exhibit systematic limitations. Vision-only saliency preserves broad spatial coverage and visually salient regions, but may discard small, text-critical regions. Text–vision saliency captures query-aligned local details, but is often sparse and biased towards late-index patches. ConsensusDrop (ours) retains complementary evidence by fusing visual and cross-modal saliency signals, preserving information required across diverse questions. Motivated by the substantial redundancy in visual tokens, a large body of work improves VLM efficiency by ranking and discarding less informative image patches. These works generally follow two paradigms: Cross-modal-only pruning 93 ; 11 ; 23 ; 49 ; 18 ; 67 ; 82 ; 81 ; 90 ; 86 ; 73 ranks patches using text–vision attention between the input prompt and image tokens inside the LLM, while Vision-only pruning 16 ; 70 ; 79 ; 77 ; 91 ; 29 ; 88 ; 66 filters tokens based on vision encoder-side saliency. While both families have demonstrated meaningful speedups and competitive accuracy under varying retention budgets 16 ; 70 ; 79 ; 77 ; 91 ; 29 ; 88 ; 66 , they suffer from non-trivial limitations that hinder their effectiveness. Cross-modal-only pruning suffers from: (i) strong positional bias from rotary embeddings that over-weights late-index patches, (ii) sparse and unstable cross-scores that can miss visually salient regions, and (iii) practical inefficiency—extracting layer attentions disables FlashAttention 14 and induces non-uniform KV caches with fragmented memory. Conversely, Vision-only pruning remains query-agnostic , dropping regions crucial to the question, especially the small, text-conditioned regions (e.g., embedded text snippets or chart elements) in document or OCR-style tasks. Moreover, vision encoders exhibit attention-sink behavior, which inflate background scores at the expense of fine-grained, query-relevant details. Additionally, several methods in both families require additional VLM training 38 ; 10 ; 89 ; 7 ; 25 ; 36 ; 66 , which imposes additional training overhead. We hypothesize that visual and cross-modal saliency are complementary and that both are essential for robust visual token selection (Fig. 1 ). Our controlled experiments confirm this hypothesis: enforcing a multimodal consensus yields more reliable importance estimates and consistently superior performance compared to unimodal pruning. However, exploiting such consensus introduces two key challenges. First, cross-modal saliency is typically computed inside the language model, forcing pruning to occur late in the inference pipeline, which incurs additional overhead and can lead to inefficient execution and KV cache fragmentation 77 . Second, visual and cross-modal signals are inherently asymmetric —one provides broad visual coverage while the other is sparse but query-specific—making naive fusion ineffective. This raises a fundamental question: how can we reconcile these complementary yet asymmetric signals to derive a principled consensus for token selection? Building upon these findings and addressing the limitations, we propose ConsensusDrop , a training-free framework for efficient visual token reduction. Specifically, we first introduce the Static Cross-Attention Probe (SCAP) , a lightweight and training-free module that extracts text–vision attention scores before they enter the language model, thereby avoiding the need to access internal LLM attention maps. Using SCAP and the vision encoder’s attention scores, we then design a novel Fuser Module that performs “multimodal” score fusion, identifying the top- K most informative visual tokens to retain. Subsequently, the remaining tokens are compressed into a dense subset of visual tokens via a novel Encoder-Guided Token Merge (EGTM) module. Empirically, ConsensusDrop achieves state-of-the-art accuracy-efficiency tradeoffs across diverse multimodal benchmarks. By pruning before LLM, it retains FlashAttention 14 and lowers KV-cache cost. The training-free design naturally supports generalization to high-resolution and video-based models (e.g., LLaVA-Next and Video-LLaVA). These results highlight that multimodal consensus is an effective method for visual token selection. Our key contributions are as follows: • We show that multimodal consensus —combining visual saliency with cross-modal cues—consistently outperforms unimodal visual token pruning. • We introduce a Static Cross-Attention Probe (SCAP) , a lightweight pre-LLM module for efficiently extracting query-aware cross-modal saliency. • To reconcile the asymmetry between visual and cross-modal cues, we propose a Fuser Module that yields a unified and robust visual token-importance ranking. • We propose Encoder-Guided Token Merging (EGTM) to compactly merge discarded tokens in the projector space under vision encoder guidance. • ConsensusDrop is training-free and plug-and-play, and achieves state-of-the-art accuracy–efficiency trade-offs across multiple open-source VLMs. 2 Related Works Vision-only Pruning. This paradigm scores tokens using vision encoder-side saliency signals to remove redundancy before the LLM. Representative methods include 70 ; 66 , 16 ; 91 ; 79 ; 29 . VisionZip 77 and VisPruner 88 achieve state-of-the-art performance by scoring visual patches via encoder attention and compressing discarded tokens through contextual merging or similarity-driven de-duplication. While effective and training-free, these methods remain inherently query-agnostic because importance is determined solely in the visual modality. Cross-modal-only Pruning. This line of work leverages text–vision cross-attention extracted from within the LLM to rank and prune patches. Approaches range from early-layer ranking 11 ; 90 and multi-stage or collaborative aggregation 73 ; 81 ; 23 ; 49 to progressive removal and dynamic skipping across LLM layers 18 ; 82 ; 67 ; 93 ; 86 ; 76 . While query-aware, extracting signals inside the LLM disables FlashAttention optimizations, fragments the KV cache, and yields sparse, prompt-biased scores that often overlook visually salient regions. ConsensusDrop overcomes these limitations by fusing query-aware cross-modal signals before the LLM with vision-side saliency, enabling training-free, plug-and-play token reduction while retaining inference efficiency. 3 Motivation 3.1 Do We Need Both Vision and Cross-modal Cues? We hypothesize that neither the vision encoder nor the language model alone provides a reliable estimate of visual token importance. To study this, we design a controlled setting where both modalities independently score the same N image patches. We first evaluate the agreement , which represents overlap between the top- K tokens derived from vision and cross-modal scores. To examine whether one modality can correct the other, we perform a simple recovery procedure: we discard the lowest-ranked tokens from one modality (the student) and replace them with the highest-ranked candidates from the other (the teacher). This controlled setup isolates the contribution of visual versus cross-modal saliency and reveals how often each modality captures information that the other misses, motivating our approach. The controlled setting reveals that (Sec. 3.3 ) (i) vision- and cross-modal saliency produce non-trivial ranking disagreements, and (ii) the signals are asymmetric : vision saliency provides a stronger base ranking, while cross-modal cues primarily contribute targeted corrections . This motivates multimodal fusion with an explicit vision bias as our default design (Sec. 4.2 ). 3.2 Problem Formulation VLM Formulation. Formally, a VLM consists of a frozen vision encoder \mathcal{E}_{v} , a projector \mathcal{P} , and a frozen LLM \mathcal{L} . Given an image \mathcal{I} and query \mathbf{T}\in\mathbb{R}^{M\times d} , the image is projected into visual tokens \tilde{\mathbf{V}}\in\mathbb{R}^{N\times d} . The model processes the concatenated sequence \mathbf{X}=[\mathbf{X}^{\text{sys}},\tilde{\mathbf{V}},\mathbf{T}] . Since N\gg M , our target is to compress \tilde{\mathbf{V}} into a compact subset before LLM inference. Scores and Selection. Vision-side scores t_{v}\in\mathbb{R}^{N} are obtained from the penultimate self-attention layer of the vision encoder using CLS-based attention. Cross-modal scores t_{c}\in\mathbb{R}^{N} are extracted from a designated transformer layer of the LLM by aggregating attention from all query tokens to each visual token. This follows the setting of visual token pruning as in FastV 11 . For this controlled study, to ensure a fixed budget, we define a Retention Ratio \rho and set the selection count K=\rho N . For each score vector, we take the top- K patch indices, denoted I_{v}(K) and I_{c}(K) . Agreement and Disagreement: We define agreement as the overlap \mathrm{agreement}(K)=|I_{v}\cap I_{c}|/K , measuring how consistently the two modalities identify important regions; the disagreement ratio is 1-\mathrm{agreement}(K) . Recovery and Correction: We perform the recovery (Sec. 3.1) by treating one set from I_{v}(K) and I_{c}(K) as the student I_{s} and the other as the teacher I_{t} . For a recovery rate r , we drop the lowest rK tokens from I_{s}(K) and replace them with the highest-ranked rK tokens from I_{t}(K) that do not already appear in the retained student tokens. Let \ell be the deepest teacher rank we must inspect to find rK such non-overlapping tokens; we define a correction rate as (K-\ell)/(K-rK) , which lies in [0,1] and reflects how strongly the teacher disagrees with the student’s ranking. 3.3 Analysis Accuracy Gains with Multimodal Fusion. We evaluate LLaVA-1.5-7B accuracy across six benchmarks (Fig. 2 ). Recovery consistently boosts accuracy at all retention ratios, confirming that joint cues estimate token importance better than either signal alone. Notably, vision scores generally outperform cross-modal scores alone. This implies an asymmetric synergy : visual cues provide the foundation, helping cross-modal cues more than vice-versa, aligning with prior vision-only pruning findings 77 ; 88 ; 29 . Takeaway. Vision saliency provides a strong base ranking, while cross-modal cues primarily correct a small, critical subset. This motivates a vision-biased consensus mechanism (Sec. 4.2 ). Figure 2 : Multimodal recovery consistently improves accuracy across benchmarks. Using LLaVA-1.5-7B, we compare token selection based on (i) cross-modal scores, (ii) vision scores, and (iii) our recovery mechanism that fuses the two. Across POPE, MME, MMBench, MMBench-CN, TextVQA, and GQA, recovery achieves higher accuracy at all retention ratios (0.25 and 0.50) and for both student configurations (visual and cross-modal). These validate that neither unimodal signal is sufficient , and that combining them provides a more reliable estimate of token importance. Disagreement-Correction-Accuracy Mechanism. Fig. 3 reveals a clear causal chain: aggressive pruning (low retention) amplifies modal disagreement, creating opportunities for valid cross-modal corrections that directly translate into accuracy gains. Essentially, disagreement acts as a signal, highlighting the specific ”blind spots” where cross-modal intervention is most effective. Takeaway. Since improvements stem from correcting specific disagreements, cross-modal cues are most valuable when applied selectively to these conflict regions rather than uniformly (Sec. 4.2 ). Figure 3 : Three-way relationship between disagreement, correction rate, and accuracy improvement. Bubble size denotes disagreement between visual and cross-modal attention; color distinguishes retention ratio ( \rho=0.25 vs. \rho=0.5 ). Higher disagreement consistently leads to higher correction, which yields greater accuracy gains. Low-retention (red) exhibit the strongest effect, confirming that aggressive pruning amplifies productive modal disagreements that consensus recovery can leverage. Correction-Rate Decay. In Fig. 4 , correction rates start high (80–90% at r=0.1 ), showing early recoveries target genuine disputed tokens. Rates drop sharply (to 40–50%) as r increases, revealing a consensus-saturation effect: only a few dropped tokens meaningfully disagree between modalities, and recovering beyond this set reintroduces largely redundant tokens. Takeaway. Since further recovery yields diminishing returns, the remaining low-saliency tokens should be merged rather than retained individually, motivating Encoder-Guided Token Merge (Sec. 4.3 ). Figure 4 : Correction-rate decay as the recovery rate r increases for POPE and GQA. Correction rates are highest at small r , where nearly all recovered tokens correspond to true multimodal disagreements. As r increases, the benefit of recovery saturates: additional tokens reintroduced are increasingly ones that both modalities already agreed upon. This reveals a strong quality–quantity tradeoff and shows that consensus-based recovery naturally prioritizes a small set of high-value corrections. What Consensus Means. Throughout this paper, we use consensus to denote agreement between vision-side and cross-modal saliency signals after reconciling their complementary roles: vision provides a strong base ranking, while cross-modal cues introduce targeted, query-dependent corrections. Rather than equal weighting, it implies a principled integration of both signals. 4 Method Figure 5 : Illustration of ConsensusDrop. Given an image–question pair, a frozen vision encoder and projector produce visual tokens, and SCAP provides query-aware cross-attention scores. A lightweight fuser combines vision and cross-modal saliency to select top- K tokens, while EGTM merges the remaining tokens into compact representations, yielding a compressed visual sequence for the LLM. Building on Sec. 3 , we present ConsensusDrop , a training-free visual token compression framework fusing vision and cross-modal saliency via three training-free components: the Static Cross-Attention Probe (SCAP), a Multimodal Fuser, and Encoder-Guided Token Merge (EGTM). In Fig. 5 , SCAP first extracts cross-modal attention before the LLM. The Fuser then combines these with vision-side saliency to select the top- K tokens, while EGTM uses the encoder’s feature geometry to compactly merge the remaining low-saliency tokens in the projector space, yielding a simplified yet information-preserving sequence (Algorithm 2 ). 4.1 Static Cross-Attention Probe (SCAP) SCAP is a lightweight, training-free probe that estimates query-conditioned saliency for each visual token before the main LLM forward pass. Concretely, SCAP replicates the input LayerNorm and multi-head self-attention module from the first decoder layer of the LLM, and applies it once to the full multimodal input sequence \mathbf{X}=[\mathbf{X}^{\text{sys}};\tilde{\mathbf{V}};\mathbf{T}] . Let S=|\mathbf{X}^{\text{sys}}|+N+L denote the total sequence length. The resulting head-averaged attention matrix \tilde{\mathbf{A}}\in\mathbb{R}^{S\times S} is \tilde{\mathbf{A}}=\frac{1}{H}\sum_{h=1}^{H}\mathrm{Softmax}\!\left(\frac{\mathbf{Q}^{(h)}(\mathbf{K}^{(h)})^{\top}}{\sqrt{d_{h}}}\right), (1) where Softmax is applied row-wise and the same causal mask as the LLM is used. We extract the text-to-vision block \mathbf{A}=\tilde{\mathbf{A}}[\mathcal{I}_{\text{text}},\mathcal{I}_{\text{vis}}]\in\mathbb{R}^{L\times N} , where \mathbf{A}_{i,j} denotes the attention from text token t_{i} to visual token \tilde{v}_{j} . We aggregate \mathbf{A} into cross-modal saliency scores \mathbf{s}^{(c)}\in\mathbb{R}^{N} using the all-token strategy by default: \mathbf{s}^{(c)}=\frac{1}{L}\sum_{i=1}^{L}\mathrm{Norm}(\mathbf{A}_{i,:}) (see Appendix D.1 for other variants). 4.2 Fuser Module Given an input image, we obtain two complementary saliency vectors over the N visual tokens: a vision-side score and a cross-modal score. From the frozen vision encoder, we extract vision-side saliency \mathbf{s}^{(v)}\in\mathbb{R}^{N} using either (i) CLS-to-patch attention or (ii) the aggregate attention received by each patch from all other patches, both computed from the penultimate self-attention layer of the vision encoder (with attention scores aggregated across heads). SCAP (Sec. 4.1 ) produces the query-conditioned cross-modal saliency \mathbf{s}^{(c)}\in\mathbb{R}^{N} . Temperature Scaling. Before fusion, both saliency vectors are converted into normalized importance distributions using temperature scaling. For modality m\in\{v,c\} , we define \hat{\mathbf{s}}^{(m)}_{j}=\frac{\left(\mathbf{s}^{(m)}_{j}\right)^{1/\tau_{m}}}{\sum_{k=1}^{N}\left(\mathbf{s}^{(m)}_{k}\right)^{1/\tau_{m}}},\qquad j=1,\dots,N, (2) where \tau_{m}>0 controls distribution sharpness. Larger temperatures yield flatter distributions, while smaller temperatures emphasize high-scoring tokens (sharper distribution). Convex Fuser. Our default Fuser combines the two normalized distributions via a convex combination: \hat{\mathbf{s}}=\alpha\,\hat{\mathbf{s}}^{(v)}+(1-\alpha)\,\hat{\mathbf{s}}^{(c)}, (3) where \alpha\in[0,1] controls the vision–cross-modal balance. Motivated by the asymmetric behavior observed in Sec. 3 , we bias fusion toward vision saliency and use \alpha>0.5 (default \alpha=0.7 ) unless otherwise specified. Tokens are ranked according to \hat{\mathbf{s}} ; the top- K indices \mathcal{I}_{\text{top}} are preserved, and the remaining \mathcal{I}_{\text{non}}=[N]\setminus\mathcal{I}_{\text{top}} are passed to EGTM for compression. The fused distribution \hat{\mathbf{s}} is only defined for this convex fuser. (See Appendix D.2 for alternative fusion mechanisms). 4.3 Encoder-Guided Token Merging (EGTM) After the Fuser selects the top- K indices \mathcal{I}_{\text{top}} , we compress the remaining R=N-K visual tokens \mathcal{I}_{\text{non}}=[N]\setminus\mathcal{I}_{\text{top}} into M merged tokens, producing a final visual sequence of length K+M . EGTM is encoder-guided : structural decisions (anchor selection and token assignment) are computed using features from the frozen vision encoder, while feature aggregation is performed in the post-projector space to maintain compatibility with the LLM. Anchor Selection via FPS. Let \mathbf{V}\in\mathbb{R}^{N\times d_{v}} denote the penultimate-layer vision encoder patch features (pre-projector), and define \mathbf{V}_{\text{non}}=\mathbf{V}[\mathcal{I}_{\text{non}}]\in\mathbb{R}^{R\times d_{v}} . We \ell_{2} -normalize \mathbf{V}_{\text{non}} and select M diverse anchor tokens using farthest point sampling (FPS) 60 , yielding a set of anchor indices \mathcal{A}\subseteq\mathcal{I}_{\text{non}} with |\mathcal{A}|=M . The remaining indices are denoted by \mathcal{U}=\mathcal{I}_{\text{non}}\setminus\mathcal{A} . Hard Assignment via Encoder Keys. Let \mathbf{K}\in\mathbb{R}^{H_{v}\times N\times d_{k}} denote the key tensors from the same penultimate vision encoder block (one per attention head). We construct a head-averaged, \ell_{2} -normalized key embedding per patch, resulting in \mathbf{G}\in\mathbb{R}^{N\times d_{k}} . Restricting to non-top- K tokens gives \mathbf{G}_{\text{non}}=\mathbf{G}[\mathcal{I}_{\text{non}}]\in\mathbb{R}^{R\times d_{k}} . Each non-anchor (merge) token u\in\mathcal{U} is assigned to the most similar anchor in this key space via a hard nearest-anchor rule: a^{\star}(u)=\arg\max_{a\in\mathcal{A}}\;\langle\mathbf{g}_{u},\mathbf{g}_{a}\rangle , where \mathbf{g}_{u} and \mathbf{g}_{a} are rows of \mathbf{G}_{\text{non}} . This dot-product is used solely for assignment , yielding a discrete partition of non-top- K tokens around the selected anchors. Merging in Projector Space. Let \tilde{\mathbf{V}}\in\mathbb{R}^{N\times d} denote the projected visual tokens fed to the LLM. For each anchor a\in\mathcal{A} , we form one merged token by averaging its own projected token with all projected tokens assigned to it: \tilde{\mathbf{v}}^{\text{merge}}_{a}=\frac{1}{|\mathcal{C}(a)|}\sum_{j\in\mathcal{C}(a)}\tilde{\mathbf{v}}_{j}, (4) where \mathcal{C}(a)=\{a\}\cup\{u\in\mathcal{U}:a^{\star}(u)=a\}. Finally, EGTM outputs the reduced visual sequence \tilde{\mathbf{V}}_{\text{final}}=\mathrm{Concat}\big(\tilde{\mathbf{V}}[\mathcal{I}_{\text{top}}],\{\tilde{\mathbf{v}}^{\text{merge}}_{a}\}_{a\in\mathcal{A}}\big)\in\mathbb{R}^{(K+M)\times d}. (5) Multimodal Design Rationale. EGTM explicitly decouples where structure is inferred from where compression is applied: the vision encoder provides a semantically meaningful metric space for selecting anchors and assigning tokens, while the final merging is performed in the LLM-aligned projector space. This design enables aggressive visual token compression while preserving the multimodal representations consumed by the language model. Method \text{VQA}^{\text{V2}} GQA \text{SQA}^{\text{IMG}} \text{VQA}^{\text{Text}} POPE MME MMB \text{MMB}^{\text{CN}} MMVet Acc Rel TTFT (ms) TPOT (ms) KV (MB) Upper bound: 576 tokens (100%) LLaVA-1.5-7B 78.5 62.0 66.8 58.2 85.9 1510.7 64.3 58.3 31.1 64.5 100% 106.4 27.7 321 Retain 192 tokens ( \downarrow 66.7\% ) ToMe 6 68.0 54.3 65.2 52.1 72.4 1148.8 60.5 50.2 27.8 56.4 87.4% — — — FastV 11 67.1 52.6 69.1 52.5 64.8 1252.1 61.0 52.4 28.5 56.7 87.9% 78.0 (1.36 \times ) 26.0 (1.07 \times ) 144 ( \downarrow 55%) SparseVLM 90 77.0 59.5 68.7 57.8 85.3 1395.4 64.1 52.1 30.2 62.7 97.2% 86.6 (1.23 \times ) 38.3 (0.72 \times ) 119 ( \downarrow 63%) VisionZip 77 76.8 59.3 68.9 57.3 85.3 1443.6 63.0 56.9 31.7 63.5 98.4% 69.5 (1.53 \times ) 25.5 (1.09 \times ) 129 ( \downarrow 60%) VisPruner 88 76.9 59.6 68.7 57.6 86.0 1468.4 63.7 57.4 33.3 64.1 99.4% 73.0 (1.46 \times ) 26.2 (1.06 \times ) 129 ( \downarrow 60%) ConsensusDrop 77.1 60.4 69.0 57.5 85.5 1476.2 63.8 58.0 33.1 64.3 99.7% 80.4 (1.32 \times ) 26.2 (1.06 \times ) 129 ( \downarrow 60%) Retain 128 tokens ( \downarrow 77.8\% ) ToMe 6 63.0 52.4 59.6 49.1 62.8 1088.4 53.3 48.8 27.2 52.3 81.1% — — — FastV 11 61.8 49.6 60.2 50.6 59.6 1208.9 56.1 51.4 28.1 53.1 82.3% 72.1 (1.48 \times ) 25.8 (1.07 \times ) 117 ( \downarrow 64%) SparseVLM 90 73.8 56.0 67.1 54.9 80.5 1376.2 60.0 51.1 30.0 60.2 93.3% 83.0 (1.28 \times ) 37.7 (0.74 \times ) 87 ( \downarrow 73%) PruMerge+ 65 74.7 57.8 67.6 54.3 81.5 1420.5 61.3 54.7 28.7 61.3 95.0% — — — VisionZip 77 75.6 57.6 68.9 56.8 83.2 1432.4 62.0 56.7 32.6 62.8 97.4% 60.2 (1.77 \times ) 25.3 (1.10 \times ) 97 ( \downarrow 70%) VisPruner 88 75.8 58.2 69.1 57.0 84.6 1461.4 62.7 57.3 33.7 63.5 98.4% 72.5 (1.47 \times ) 26.2 (1.06 \times ) 97 ( \downarrow 70%) ConsensusDrop 76.1 60.2 69.3 57.1 85.1 1467.7 63.2 58.1 33.8 64.0 99.2% 70.6 (1.51 \times ) 25.8 (1.07 \times ) 97 ( \downarrow 70%) Retain 64 tokens ( \downarrow 88.9\% ) ToMe 6 57.1 48.6 50.0 45.3 52.5 922.3 43.7 38.9 24.1 45.1 69.9% — — — FastV 11 55.0 46.1 51.1 47.8 48.0 1019.6 48.0 42.7 25.8 46.2 71.6% 71.4 (1.50 \times ) 25.9 (1.07 \times ) 96 ( \downarrow 70%) SparseVLM 90 68.2 52.7 62.2 51.8 75.1 1221.1 56.2 46.1 23.3 55.2 85.6% 84.0 (1.27 \times ) 38.1 (0.73 \times ) 55 ( \downarrow 83%) PruMerge+ 65 67.4 54.9 68.6 53.0 77.4 1198.2 59.3 51.0 25.9 57.5 89.1% — — — VisionZip 77 72.4 55.1 69.0 55.5 77.0 1365.6 60.1 55.4 31.7 60.5 93.8% 58.4 (1.82 \times ) 25.5 (1.09 \times ) 65 ( \downarrow 80%) VisPruner 88 72.7 55.4 69.1 55.8 80.4 1369.9 61.3 55.1 32.3 61.2 94.9% 72.0 (1.48 \times ) 25.7 (1.08 \times ) 65 ( \downarrow 80%) ConsensusDrop 73.1 56.2 70.2 56.4 79.8 1372.9 61.7 55.8 31.2 61.4 95.2% 64.7 (1.64 \times ) 25.2 (1.10 \times ) 65 ( \downarrow 80%) Retain 32 tokens ( \downarrow 94.4\% ) ToMe 6 46.8 43.6 41.4 38.3 39.0 828.4 31.6 28.1 17.3 36.4 56.4% — — — FastV 11 43.4 41.5 42.6 42.5 32.5 884.6 37.8 33.2 20.7 37.6 58.3% 66 (1.61 \times ) 26.3 (1.05 \times ) 81 ( \downarrow 75%) SparseVLM 90 58.6 48.3 57.3 46.1 67.9 1046.7 51.4 40.6 18.6 49.0 76.0% 77.8 (1.37 \times ) 36.7 (0.75 \times ) 48 ( \downarrow 85%) PruMerge+ 65 54.9 51.1 68.5 50.6 70.9 940.8 56.8 47.0 21.4 52.0 80.6% — — — VisionZip 77 67.1 51.8 68.8 53.1 68.7 1247.4 57.7 50.3 25.5 56.2 87.1% 54.3 (1.96 \times ) 26.1 (1.06 \times ) 49 ( \downarrow 85%) VisPruner 88 67.7 52.2 69.2 53.9 72.7 1271.0 58.4 52.7 28.8 57.7 89.5% 71.5 (1.49 \times ) 26.4 (1.05 \times ) 49 ( \downarrow 85%) ConsensusDrop 67.8 53.2 70.1 54.7 73.1 1275.8 59.0 54.1 30.1 58.4 90.5% 60.0 (1.77 \times ) 25.1 (1.10 \times ) 49 ( \downarrow 85%) Table 1 : Performance comparison on LLaVA-1.5-7B across several retention settings. Acc is the average accuracy; Rel is relative performance vs. baseline. TTFT = Time To First Token with speedup in parentheses; TPOT = Time Per Output Token with speedup; KV = KV cache memory with reduction percentage. Bold indicates best; underline indicates second-best. 5 Experiments 5.1 Experimental setup Datasets. We evaluate ConsensusDrop on 12 benchmarks across three distinct categories: (1) General VQA : VQAv2 21 , GQA 27 , ScienceQA-IMG 52 , and TextVQA 68 ; (2) Multimodal Understanding : POPE 39 , MME 20 , MMBench/CN 48 , and MM-Vet 84 ; and (3) Video QA : TGIF-QA 28 , MSVD-QA 74 , and MSRVTT-QA 74 . All experiments follow official evaluation protocols. Model Architectures. We apply ConsensusDrop to representative VLMs: (1) the LLaVA Family (main experiments): LLaVA-1.5 43 , LLaVA-NeXT 44 (high-resolution), and Video-LLaVA 40 (video understanding); and (2) Other Open-source VLMs : Qwen-VL 4 , InternVL 12 , and CogVLM 71 . Evaluation Setting. We follow the official inference hyperparameters and decoding settings for each model. For ConsensusDrop, we use the Convex Fuser by default, with \tau_{c}=\tau_{m}=1 and \alpha=0.7 , with EGTM enabled. Comparison Methods. As pruning baselines, we consider ToMe 6 , FastV 11 , SparseVLM 90 , LLaVA-PruMerge 65 , VisionZip 77 , and VisPruner 88 . These methods cover a variety of pruning strategies applied at different stages of VLMs, providing a comprehensive and fair comparison. 5.2 Main Results Table 1 presents the performance of ConsensusDrop on LLaVA-1.5-7B across four retention budgets (192, 128, 64, and 32 visual tokens out of 576). ConsensusDrop consistently achieves the strongest accuracy–efficiency trade-off across all budgets, preserving 99.7% , 99.2% , 95.2% , and 90.5% of baseline performance, respectively. As compression becomes more aggressive, the advantage of ConsensusDrop becomes decisive. At 32 tokens ( \downarrow 94.4%), it achieves the best performance across all benchmarks, substantially outperforming both vision-only and cross-modal-only methods while retaining 90.5% of baseline accuracy. This result highlights the importance of multimodal consensus under extreme token reduction, where selecting the right visual tokens becomes critical. From an efficiency perspective, ConsensusDrop consistently improves over cross-modal-only methods such as FastV 11 and SparseVLM 90 in both TTFT and KV cache footprint, while remaining competitive with vision-only methods. VisionZip 77 achieves the lowest latency across settings; however, ConsensusDrop delivers substantially higher accuracy at comparable token budgets, yielding a more favorable accuracy–latency trade-off overall. Finally, while SparseVLM 90 has the smallest analytical KV cache size, its LLM-internal stage-wise pruning leads to non-uniform KV cache sizes across layers, which can result in memory fragmentation not reflected in Table 1 . In contrast, ConsensusDrop performs pruning prior to LLM inference, leading to a compact and uniform KV cache. 5.3 ConsensusDrop with Higher Resolution Inputs Across high-resolution inputs in LLaVA-NeXT-7B 45 , where the visual sequence expands to 2880 tokens, ConsensusDrop consistently matches or outperforms the strongest pruning baselines under identical token budgets (Tab. 2 ). At 640 tokens ( \downarrow 77.8\% ), it preserves 99.0% of full-model performance, slightly exceeding VisPruner and maintaining strong accuracy across GQA, POPE, and MME. With more aggressive pruning to 320 ( \downarrow 88.9\% ) and 160 tokens ( \downarrow 94.4\% ), ConsensusDrop retains 93.9% and 87.1% relative performance, respectively, outperforming VisionZip, VisPruner, and cross-modal-only methods by clear margins. These results show that multimodal consensus remains effective even under dense tokenization, enabling aggressive compression while preserving LLM-aligned visual evidence. 5.4 ConsensusDrop with Video Understanding Tab. 3 evaluates ConsensusDrop on Video-LLaVA across three video QA benchmarks, a setting with heavy temporal redundancy from continuous frames (8 frames at 224 resolution, 2048 tokens). Under identical reduction ratios, ConsensusDrop consistently matches or outperforms prior methods. At 455 tokens ( \downarrow 77.8\% ), it achieves the best average accuracy/score (48.6/3.32), matching the full model and slightly exceeding VisPruner and FastV. With more aggressive pruning to 227 ( \downarrow 88.9\% ) and 114 tokens ( \downarrow 94.4\% ), ConsensusDrop maintains the strongest average performance, retaining roughly 92% of full-model accuracy and outperforming competing baselines. Figure 6 : Global accuracy–efficiency trade-offs on LLaVA-1.5-7B. ConsensusDrop consistently lies on or above the Pareto frontier compared to vision-only (VisPruner) and cross-modal-only (SparseVLM) pruning across token budgets \{192,128,64,32\} . 5.5 ConsensusDrop on other VLM Architectures Method \text{VQA}^{\text{V2}} GQA \text{VQA}^{\text{Text}} POPE MME Acc. Rel. Upper bound: all 2880 visual tokens ( \mathbf{100\%} ) LLaVA-NeXT-7B 81.2 62.9 59.6 86.3 1513.8 73.1 100.0% Retain 640 tokens ( \downarrow\mathbf{77.8\%} ) FastV 11 78.9 60.4 58.4 83.1 1477.3 70.9 97.0% SparseVLM 90 78.2 59.1 56.2 80.9 1456.3 69.4 94.9% VisionZip 77 79.2 60.1 58.5 82.2 1468.4 70.7 96.7% VisPruner 88 79.8 61.6 59.3 85.9 1480.7 72.1 98.6% ConsensusDrop 79.9 62.1 59.7 86.1 1483.9 72.4 99.0% Retain 320 tokens ( \downarrow\mathbf{88.9\%} ) FastV 11 71.9 55.9 55.7 71.7 1282.9 63.9 87.4% SparseVLM 90 71.4 56.5 52.4 73.5 1342.7 64.2 87.8% VisionZip 77 74.2 58.1 55.3 75.0 1348.8 66.0 90.3% VisPruner 88 75.7 58.4 57.6 80.4 1370.1 68.1 93.2% ConsensusDrop 76.1 59.1 58.0 81.5 1372.8 68.7 93.9% Retain 160 tokens ( \downarrow\mathbf{94.4\%} ) FastV 11 61.8 49.8 51.9 51.7 1079.5 53.8 73.6% SparseVLM 90 62.2 50.2 45.1 54.6 1167.1 54.1 74.0% VisionZip 77 67.3 54.3 54.7 59.4 1239.7 59.5 81.4% VisPruner 88 70.6 54.7 56.0 72.9 1226.0 63.1 86.3% ConsensusDrop 71.0 55.3 57.2 73.2 1232.1 63.7 87.1% Table 2 : Performance comparison of pruning methods on LLaVA-NeXT-7B. Acc. denotes the aggregated score across benchmarks, and Rel. represents the percentage of performance retained at the corresponding reduction ratio relative to the full-token LLaVA-NeXT-7B model. Method TGIF-QA MSVD-QA MSRVTT-QA Average Acc. Score Acc. Score Acc. Score Acc. Score Upper bound: all 2048 visual tokens ( \mathbf{100\%} ) Video-LLaVA 19.8 2.53 70.5 3.93 57.5 3.50 49.3 3.32 Retain 455 tokens ( \downarrow\mathbf{77.8\%} ) FastV 11 19.2 2.50 69.1 3.91 54.4 3.42 47.6 3.28 VisPruner 88 18.4 2.49 70.2 3.95 56.7 3.50 48.4 3.31 ConsensusDrop 18.6 2.51 70.3 3.94 56.8 3.52 48.6 3.32 Retain 227 tokens ( \downarrow\mathbf{88.9\%} ) FastV 11 14.3 2.42 68.9 3.90 53.0 3.40 45.4 3.24 VisPruner 88 15.9 2.41 69.3 3.92 55.6 3.45 46.9 3.26 ConsensusDrop 16.1 2.43 69.5 3.94 55.8 3.43 47.1 3.27 Retain 114 tokens ( \downarrow\mathbf{94.4\%} ) FastV 11 10.6 2.29 64.1 3.78 52.4 3.39 42.4 3.15 VisPruner 88 14.1 2.35 65.4 3.79 54.1 3.41 44.5 3.18 ConsensusDrop 14.4 2.38 66.7 3.82 54.4 3.45 45.2 3.22 Table 3 : Video understanding performance with Video-LLaVA across three video question answering benchmarks. Performance is evaluated on the first 1,000 samples from each benchmark, and gpt-3.5-turbo is used for scoring as in 11 . Method \text{VQA}^{\text{V2}} GQA \text{SQA}^{\text{IMG}} \text{VQA}^{\text{Text}} Acc. Rel. Upper bound: all 256 visual tokens ( \mathbf{100\%} ) Qwen-VL-7B 78.8 59.3 67.1 63.8 67.3 100.0% Retain 128 tokens ( \downarrow\mathbf{50\%} ) FastV 11 76.5 56.9 65.3 58.2 64.2 95.4% VisPruner 88 77.4 57.8 65.9 59.6 65.2 96.9% ConsensusDrop 78.1 58.9 66.4 61.1 66.1 98.2% Upper bound: all 576 visual tokens ( \mathbf{100\%} ) InternVL-Chat-13B 79.3 62.9 66.3 57.0 66.4 100.0% Retain 144 tokens ( \downarrow\mathbf{75\%} ) FastV 11 74.1 58.2 66.6 55.6 63.6 95.8% VisPruner 88 76.7 60.2 67.3 55.3 64.9 97.7% ConsensusDrop 76.9 60.5 67.9 56.1 65.4 98.5% Upper bound: all 1225 visual tokens ( \mathbf{100\%} ) CogVLM-chat-1.1-17B 80.9 58.2 68.4 69.2 69.2 100.0% Retain 123 tokens ( \downarrow\mathbf{90\%} ) FastV 11 74.2 40.3 63.5 41.9 55.0 79.5% VisPruner 88 74.6 48.4 68.2 59.6 62.7 90.6% ConsensusDrop 75.1 49.3 69.6 59.5 63.4 91.6% Table 4 : Performance of pruning methods on different VLM architectures. Acc. denotes the average accuracy across benchmarks, and Rel. represents the percentage of performance retained. We further evaluate ConsensusDrop on three representative VLM backbones with distinct vision–language interfaces: Qwen-VL-7B 4 , InternVL-Chat-13B 12 , and CogVLM-chat-1.1-17B 71 (Tab. 4 ). Across all token budgets, ConsensusDrop consistently outperforms the cross-modal-only pruning baseline FastV 11 and yields small but reliable gains over VisPruner in average accuracy. For Qwen-VL at 128/256 tokens ( \downarrow 50\% ), ConsensusDrop retains 98.2% of full performance (Acc. 66.1 vs. 67.3), improving over VisPruner (96.9%). On InternVL at 144/576 tokens ( \downarrow 75\% ), it again achieves the best retained performance (98.5%) and the highest Acc. (65.4). Under the most extreme compression on CogVLM (123/1225 tokens, \downarrow 90\% ), ConsensusDrop preserves 91.6% of the original accuracy, exceeding both FastV (79.5%) and VisPruner (90.6%). Figure 7 : Per-dataset accuracy vs. TTFT trade-offs. Across datasets, ConsensusDrop preserves higher accuracy at comparable or higher TTFT speedups, indicating consistent efficiency gains. 5.6 Efficiency Analysis We evaluate the accuracy–efficiency trade-off of token pruning on LLaVA-1.5-7B, focusing on TTFT (prefill latency), TPOT (decode latency), and KV cache memory. Latency. As the token budget decreases, ConsensusDrop achieves substantial TTFT speedups while maintaining strong accuracy retention. At aggressive compression (32 tokens, \downarrow 94.4%), it retains 90.5% of baseline accuracy with a 1.77 \times TTFT speedup, outperforming both baselines. Memory. By pruning tokens prior to LLM inference, ConsensusDrop yields uniform KV cache layouts and consistent memory savings, reducing KV usage by up to 85% at extreme compression. Pareto Optimality. Fig. 6 shows that ConsensusDrop consistently achieves a more favorable accuracy–latency–memory trade-off than VisPruner and SparseVLM, with the advantage most pronounced at low token budgets where token selection quality is critical. Across Datasets. Fig. 7 confirms that these gains generalize across multiple benchmarks, with ConsensusDrop remaining closer to the Pareto frontier on all evaluated datasets. 6 Conclusion ConsensusDrop is a training-free, plug-and-play framework that exploits multimodal consensus to retain the most informative visual tokens in VLMs. By fusing both vision and cross-modal saliency, it consistently outperforms prior pruning methods under aggressive token reduction. Appendix A Related Works Vision Language Models. Large Language Models (LLMs) 57 ; 3 ; 72 ; 56 ; 55 ; 58 ; 1 ; 92 have driven rapid progress in multimodal reasoning, motivating the development of Vision Language Models (VLMs) for image and video understanding 46 ; 83 ; 19 ; 47 . A standard VLM 46 ; 45 ; 42 pairs a pre-trained vision encoder such as CLIP or SigLIP 64 ; 87 with a lightweight projector 46 that maps visual patches into the LLM embedding space. Although modern VLMs achieve strong performance across diverse tasks 63 ; 13 ; 80 , they still struggle on fine-grained perception tasks 59 and are prone to visual hallucination 37 ; 69 ; 8 . Efforts to improve visual grounding have included higher input resolutions 53 ; 22 and multi-encoder designs 13 ; 51 . Meanwhile, video understanding and OCR/document tasks require processing many frames or extremely high-resolution inputs 15 ; 24 ; 26 ; 50 , further increasing visual token counts. This results in VLMs processing hundreds to thousands of visual tokens per input. Visual tokens dominate prefill computation, scaling quadratically with token count, while inflating KV-cache memory and decode-time FLOPs, which scale linearly. Consequently, reducing redundant visual tokens has become essential for practical and scalable VLM deployment 30 ; 62 . Vision-only Pruning. One line of efficient VLM work prunes visual tokens directly from the vision encoder by leveraging vision encoder-side saliency signals. These approaches typically score patches using self-attention or related visual cues and remove low-importance tokens before they reach the language model. Representative methods include 70 ; 66 , 16 ; 91 ; 79 ; 29 . VisionZip 77 and VisPruner 88 achieve state-of-the-art performance by scoring visual patches via encoder attention and compressing discarded tokens through contextual merging or similarity-driven de-duplication. While effective and training-free, these methods remain inherently query-agnostic because importance is determined solely in the visual modality. Cross-Modal-only Pruning. Another line of work prunes visual tokens using text-vision cross-attention from within the LLM. FastV 11 ranks image tokens using early-layer cross-attention to textual queries, while SparseVLM 90 extends this idea with sparsified attention patterns. Subsequent works explore deeper or multi-stage cross-attention signals, including pyramid-based reduction 73 , training-free pruning via attention aggregation 81 ; 23 , and multi-encoder collaborative pruning 49 . Other approaches adapt attention-based importance for dynamic skipping or progressive removal in LLM layers 18 ; 82 ; 67 ; 93 ; 86 ; 76 . Although effective, these methods extract cross-attention inside the LLM, disabling FlashAttention, fragmenting the KV cache, and yielding only prompt-dependent sparsity that frequently overlooks salient visual regions. Training-based Token Pruning. A separate line of work integrates token pruning into VLM training, where pruning modules or compression adapters are optimized jointly with the model 7 ; 25 ; 10 ; 36 ; 38 ; 89 . These can maintain robustness under aggressive compression, but require additional training and are not plug-and-play. ConsensusDrop overcomes these limitations by extracting query-aware cross-modal signals before the LLM and fusing them with vision-side saliency, enabling training-free, plug-and-play token reduction that preserves both global visual context and query-critical patch tokens while retaining inference efficiency. Appendix B Hypothesis Evaluation Setting This appendix provides implementation details for the controlled study in Section 3 . Our goal is to compute agreement, disagreement, and recovery behavior between vision-side and cross-modal saliency signals. B.1 Experimental Setting Setup. All hypothesis experiments are run with LLaVA-1.5-7B 46 ; 42 and CLIP-ViT-L/14 at 336\times 336 , yielding N=576 visual tokens (a 24\times 24 grid). Experiments use a single NVIDIA RTX 6000 Ada GPU (48 GB). Visual Token Pruning via FastV. Following FastV 11 , we perform early visual token pruning within the LLM decoder. Let K denote the 1-indexed layer after which tokens are pruned. Attention scores are extracted from layer K , and the pruned token set is forwarded to layer K{+}1 . In all experiments, we set K=2 , so that attention is computed at the second decoder layer and pruning takes effect before the third layer. Retention Ratios & Benchmarks. We test \rho\in\{0.25,0.50\} (retain K_{\text{tok}}=\lfloor\rho N\rfloor\in\{144,288\} tokens) on POPE 39 , MME 19 , MMBench 48 , MMBench-CN 48 , TextVQA 68 , and GQA 27 . We evaluate recovery rates r\in\{0.1,0.3,0.5\} . For a given retention budget of K_{\text{tok}} visual tokens, we replace the bottom \lfloor rK_{\text{tok}}\rfloor tokens selected by the student modality with tokens drawn from the teacher modality’s top- K_{\text{tok}} set. The swapped tokens are required to be non-overlapping with the retained student tokens, and higher-ranked teacher tokens are given priority during the replacement process. B.2 Student–Teacher Recovery Mechanism We formalize the student–teacher recovery mechanism used in our hypothesis evaluation, also matching the recovery fuser in Section 4.2 . Saliency Signals. For each image, we compute two saliency vectors over the N visual tokens: (i) vision-side scores \mathbf{s}^{(v)}\in\mathbb{R}^{N} extracted from the vision encoder, and (ii) cross-modal scores \mathbf{s}^{(c)}\in\mathbb{R}^{N} extracted from the LLM decoder. Vision-Side Scores. By default, we use CLS-to-patch attention ( t_{v}^{\text{cls}} ) from the penultimate self-attention layer of the vision encoder. Cross-Modal Scores. Cross-modal saliency is extracted from decoder layer K (1-indexed), i.e., the last layer before token pruning is applied. By default, we use t_{\text{cross}}^{\text{last}} , corresponding to the attention from the final text token to the image tokens. Student/Teacher Assignment. We assign one modality as student and the other as teacher : \mathbf{s}^{(S)}\in\{\mathbf{s}^{(c)},\mathbf{s}^{(v)}\} and \mathbf{s}^{(T)} is the other. Recovery via Swap. Given retention ratio \rho and swap rate r , define K_{\text{tok}}=\lfloor\rho N\rfloor,\qquad M_{\text{tok}}=\lfloor rK_{\text{tok}}\rfloor. Recovery keeps the student’s top (K_{\text{tok}}-M_{\text{tok}}) tokens, and then swaps in M_{\text{tok}} high-ranked teacher tokens that are not already selected by the student. This implements a minimal intervention: the teacher only contributes where it disagrees with the student’s retained set. Algorithm 1 Student–Teacher Recovery 1: Student scores \mathbf{s}^{(S)}\in\mathbb{R}^{N} , teacher scores \mathbf{s}^{(T)}\in\mathbb{R}^{N} , retention ratio \rho , swap rate r 2: K_{\text{tok}}\leftarrow\lfloor\rho N\rfloor,\quad M_{\text{tok}}\leftarrow\lfloor rK_{\text{tok}}\rfloor 3: \mathcal{I}_{S}\leftarrow\textsc{ArgsortDesc}(\mathbf{s}^{(S)})[1:K_{\text{tok}}] \triangleright student top- K_{\text{tok}} 4: \mathcal{I}_{T}\leftarrow\textsc{ArgsortDesc}(\mathbf{s}^{(T)})[1:K_{\text{tok}}] \triangleright teacher top- K_{\text{tok}} 5: \mathcal{I}_{1}\leftarrow\mathcal{I}_{S}[1:K_{\text{tok}}-M_{\text{tok}}] \triangleright keep student top 6: \mathcal{I}_{2}\leftarrow[\,] 7: for i=1 to K_{\text{tok}} do 8: if |\mathcal{I}_{2}|=M_{\text{tok}} then break 9: end if 10: if \mathcal{I}_{T}[i]\notin\mathcal{I}_{1} then 11: append \mathcal{I}_{T}[i] to \mathcal{I}_{2} 12: end if 13: end for 14: \mathcal{I}_{\text{final}}\leftarrow\textsc{Concat}(\mathcal{I}_{1},\mathcal{I}_{2}) \triangleright |\mathcal{I}_{\text{final}}|=K_{\text{tok}} 15: return \mathcal{I}_{\text{final}} Agreement/Disagreement. Let \mathcal{I}_{S} and \mathcal{I}_{T} be the student/teacher top- K_{\text{tok}} sets. We compute: \text{Agreement}=\frac{|\mathcal{I}_{S}\cap\mathcal{I}_{T}|}{K_{\text{tok}}},\qquad\text{Disagreement}=\frac{|\mathcal{I}_{S}\setminus\mathcal{I}_{T}|}{K_{\text{tok}}}. Correction Factor. Let c be the smallest prefix length such that the teacher prefix contains M_{\text{tok}} tokens not in \mathcal{I}_{1} , i.e., |\{\mathcal{I}_{T}[1{:}c]\setminus\mathcal{I}_{1}\}|=M_{\text{tok}} . We define the correction factor as \text{CF}=\frac{K_{\text{tok}}-c}{K_{\text{tok}}-M_{\text{tok}}}. (6) Intuitively, when c is small, higher ranked teacher tokens are swapped with the student token set. This implies that tokens ranked important by the teacher were missed by the student – leading to “stronger” or “higher” correction through recovery. Appendix C ConsensusDrop and EGTM Algorithms Discussion. Algorithms 2 and 3 summarize the full ConsensusDrop inference pipeline and its encoder-guided token merging component. ConsensusDrop first identifies a compact set of salient visual tokens via multimodal consensus, then compresses the remaining visual information through EGTM to preserve contextual coverage under a fixed token budget. Importantly, both components are modular: the consensus stage is agnostic to the specific saliency estimator (e.g., SCAP), and EGTM operates purely in encoder space while merging in the LLM projector space. In the main paper, we focus on empirical behavior and efficiency gains enabled by this design; the algorithms are provided here for completeness and reproducibility. Algorithm 2 ConsensusDrop: Multimodal Consensus for Visual Token Reduction 1: Image \mathcal{I} , query \mathcal{Q} , Vision encoder \mathcal{E}_{v} , projector \mathcal{P} , LLM \mathcal{L} , Top- K budget K , merge budget M 2: \mathbf{s}^{(v)},\mathbf{V},\mathbf{K}\leftarrow\mathcal{E}_{v}(\mathcal{I}) \triangleright vision scores, tokens and keys 3: \tilde{\mathbf{V}}\leftarrow\mathcal{P}(\mathbf{V}) \triangleright project to text space 4: \mathbf{T}\leftarrow\textsc{Tokenize}(\mathcal{Q}) \triangleright LLM tokenizer 5: \mathbf{s}^{(c)}\leftarrow\textsc{SCAP}(\mathbf{T},\tilde{\mathbf{V}}) \triangleright cross-modal scores 6: \mathbf{s},\mathcal{I}_{\text{top}}\leftarrow\textsc{Fuser}(\mathbf{s}^{(v)},\mathbf{s}^{(c)},K) \triangleright fused scores and top- K indices 7: \mathcal{I}_{\text{non}}\leftarrow[1,\dots,N]\setminus\mathcal{I}_{\text{top}} \triangleright non-top- K indices 8: \tilde{\mathbf{V}}_{\text{top}}\leftarrow\tilde{\mathbf{V}}[\mathcal{I}_{\text{top}}] 9: \tilde{\mathbf{V}}_{\text{merge}}\leftarrow\textsc{EGTM}(\tilde{\mathbf{V}},\mathbf{V},\mathbf{K},\mathcal{I}_{\text{non}},M) \triangleright merge non-top 10: \tilde{\mathbf{V}}_{\text{final}}\leftarrow\textsc{Concat}(\tilde{\mathbf{V}}_{\text{top}},\tilde{\mathbf{V}}_{\text{merge}}) 11: y\leftarrow\mathcal{L}(\tilde{\mathbf{V}}_{\text{final}},\mathbf{T}) \triangleright LLM inference 12: return y \triangleright final output Algorithm 3 Encoder-Guided Token Merging (EGTM) 1: Projected visual tokens \tilde{\mathbf{V}}\in\mathbb{R}^{N\times d} , Encoder patch features \mathbf{V}\in\mathbb{R}^{N\times d_{v}} , Encoder keys \mathbf{K}\in\mathbb{R}^{H_{v}\times N\times d_{k}} , Top- K indices \mathcal{I}_{\text{top}} , merge budget M 2: \mathcal{I}_{\text{non}}\leftarrow[N]\setminus\mathcal{I}_{\text{top}} 3: \mathbf{V}_{\text{non}}\leftarrow\textsc{Normalize}(\mathbf{V}[\mathcal{I}_{\text{non}}]) 4: \mathcal{A},\mathcal{U}\leftarrow\textsc{FPS}(\mathbf{V}_{\text{non}},M) \triangleright |\mathcal{A}|=M , \mathcal{U}=\mathcal{I}_{\text{non}}\setminus\mathcal{A} 5: \mathbf{G}\leftarrow\textsc{Normalize}\!\left(\frac{1}{H_{v}}\sum_{h=1}^{H_{v}}\mathbf{K}^{(h)}\right) 6: \mathbf{G}_{\text{non}}\leftarrow\mathbf{G}[\mathcal{I}_{\text{non}}] 7: for each u\in\mathcal{U} do 8: a^{\star}(u)\leftarrow\arg\max_{a\in\mathcal{A}}\langle\mathbf{g}_{u},\mathbf{g}_{a}\rangle 9: end for 10: for each a\in\mathcal{A} do 11: \mathcal{C}(a)\leftarrow\{a\}\cup\{u\in\mathcal{U}:a^{\star}(u)=a\} 12: \tilde{\mathbf{v}}^{\text{merge}}_{a}\leftarrow\frac{1}{|\mathcal{C}(a)|}\sum_{j\in\mathcal{C}(a)}\tilde{\mathbf{v}}_{j} 13: end for 14: \tilde{\mathbf{V}}_{\text{final}}\leftarrow\mathrm{Concat}(\tilde{\mathbf{V}}[\mathcal{I}_{\text{top}}],\{\tilde{\mathbf{v}}^{\text{merge}}_{a}\}_{a\in\mathcal{A}}) 15: return \tilde{\mathbf{V}}_{\text{final}}\in\mathbb{R}^{(K+M)\times d} Appendix D Method and Implementation Details D.1 SCAP Aggregation Strategies In the main text (Sec. 4.1 ), we employ all-token aggregation as the default strategy to convert the raw text-to-vision attention block \mathbf{A}\in\mathbb{R}^{L\times N} into the final cross-modal saliency vector \mathbf{s}^{(c)}\in\mathbb{R}^{N} . Here, we formally define the three variants explored in our ablation study. Recall that \mathbf{A}_{i,j} denotes the attention weight from the i -th text token to the j -th visual token, and \mathrm{Norm}(\mathbf{z})=\mathbf{z}/(\sum_{k}z_{k}+\epsilon) denotes row-wise normalization. All-Token Aggregation (Default). We average the normalized attention scores across all L query tokens. This strategy captures the global semantic alignment between the visual features and the entire textual instruction: \mathbf{s}^{(c)}_{\text{all}}=\frac{1}{L}\sum_{i=1}^{L}\mathrm{Norm}(\mathbf{A}_{i,:}). (7) Last-Token Aggregation. This strategy relies solely on the final text token (typically the instruction terminator or separator), hypothesizing that the causal attention mechanism aggregates the full context into the last position: \mathbf{s}^{(c)}_{\text{last}}=\mathrm{Norm}(\mathbf{A}_{L,:}). (8) Max-Token Aggregation. This strategy selects the maximum attention score received by a visual token from any text token. It prioritizes peak alignment signals (e.g., a specific keyword strongly attending to a patch) regardless of the other tokens: \mathbf{s}^{(c)}_{\text{max}}=\mathrm{Norm}\left(\max_{i=1}^{L}\mathbf{A}_{i,:}\right). (9) Figure 8 : Fuser module designs. The Fuser module has two variants: Convex and Recovery . Left: the Convex Fuser temperature-normalizes vision and cross-modal scores and forms a convex combination with weight \alpha to obtain a unified importance distribution. Right: the Recovery Fuser treats cross-modal scores as the student and vision scores as the teacher, selecting top- K tokens from the student and recovering a small set of high-ranked teacher tokens based on a recovery rate. D.2 Fuser Strategies In Sec. 4.2 , we introduced the Convex Fuser as our primary method. Here, we describe the alternative Recovery Fuser and visualize both designs. Both fusion strategies are illustrated in Figure 8 . Recovery Fuser. We additionally consider a recovery-based fuser inspired by the controlled analysis in Sec. 3 . One modality is treated as a student and the other as a teacher . Starting from the student’s top- K tokens, we drop the lowest rK tokens and replace them with the highest-ranked, non-overlapping tokens from the teacher, where r\in[0,1] is a recovery rate. This mechanism operates directly on ranked indices rather than fused distributions. Unless stated otherwise, we use the convex fuser as the default and employ recovery fusion primarily to validate the analysis in Sec. 3 . Appendix E Experimental Setup and Additional Details E.1 Datasets We evaluate ConsensusDrop on a total of 13 widely used benchmarks, comprising 10 image-based benchmarks and 3 video-based benchmarks. These datasets cover a broad range of vision–language capabilities, including visual question answering, multimodal reasoning, hallucination detection, and video understanding. Unless otherwise stated, all inference protocols and evaluation metrics follow the standard settings used in LLaVA-1.5 46 ; 42 and Video-LLaVA 40 . E.1.1 Image Benchmarks We conduct experiments on 10 image benchmarks that are commonly used to evaluate large vision–language models. These include four visual question answering datasets and six multimodal reasoning and robustness benchmarks. VQAv2 21 . VQAv2 evaluates open-ended visual question answering with a strong emphasis on reducing language bias. The dataset consists of images from MSCOCO 41 , where each question is paired with multiple images corresponding to different answers. We report accuracy on the test-dev split, which contains 107,394 image–question pairs, using the standard evaluation protocol. GQA 27 . GQA focuses on compositional reasoning and structured visual understanding. Questions are generated from scene graphs derived from Visual Genome 32 , ensuring that each question corresponds to an explicit semantic reasoning path. We evaluate accuracy on the test-dev split, which contains 12,578 image–question pairs. ScienceQA (SQA-IMG) 52 . ScienceQA evaluates scientific reasoning using multiple-choice questions spanning natural science, social science, and language topics. We use the image-based subset (SQA-IMG), which contains 2,017 image–question pairs from the test set, following prior work 88 ; 77 ; 90 . TextVQA 68 . TextVQA measures a model’s ability to reason over textual content embedded in images, such as signs and product labels. Images are sourced primarily from Open Images 31 . We report accuracy on the validation set, which contains 5,000 image–question pairs. POPE 39 . POPE evaluates object hallucination in vision-language models using binary questions about object presence. Images are drawn from MSCOCO 41 . We report the average F1 score across three evaluation splits, following the standard protocol, over 8,910 image–question pairs. MME 20 . MME provides a comprehensive evaluation of multimodal perception and cognition across 14 subtasks, including OCR, object recognition, and fine-grained visual understanding. All questions are binary judgment tasks. We report the perception score, following prior work 88 , over 2,374 image–question pairs. MMBench and MMBench-CN 48 . MMBench evaluates multimodal perception and reasoning using multiple-choice questions organized across hierarchical capability levels. We evaluate both the English (MMBench) and Chinese (MMBench-CN) versions, which contain 4,377 and 4,329 image–question pairs respectively. MM-Vet 84 . MM-Vet focuses on integrated vision–language capabilities such as OCR, spatial reasoning, knowledge grounding, and mathematical reasoning. The benchmark includes 218 carefully curated image–question pairs and uses a GPT-based evaluator to score model outputs. SEEDBench 35 . SEEDBench is a large-scale multimodal benchmark designed to evaluate fine-grained visual understanding and reasoning in both images and videos. It consists of approximately 19K multiple-choice questions annotated by human assessors and spans 12 distinct evaluation aspects, covering object recognition, spatial understanding, temporal reasoning, and cross-modal comprehension. In this work, we use SEEDBench exclusively for ablation studies to analyze the behavior of different token selection and fusion components under controlled settings. E.1.2 Video Benchmarks To assess performance under higher visual redundancy, we also evaluate on three video question answering benchmarks used in Video-LLaVA 40 . Evaluation follows the Video-ChatGPT protocol 54 , using gpt-3.5-turbo as the scoring assistant. We follow FastV 11 and evaluate on the first 1,000 samples of each benchmark. TGIF-QA 28 . TGIF-QA extends visual question answering to animated GIFs, requiring models to reason over both spatial and temporal information. We evaluate performance on the Frame QA task subset. MSVD-QA 74 . MSVD-QA is built on the Microsoft Research Video Description corpus 9 , with question–answer pairs derived from video captions. The dataset contains 1,970 video clips and over 50K QA pairs. MSRVTT-QA 74 . MSRVTT-QA is based on the MSRVTT dataset 75 , which contains more diverse and complex video scenes than MSVD 9 . The benchmark includes 10K video clips and 243K question–answer pairs. E.2 Model Architectures We evaluate ConsensusDrop across representative state-of-the-art vision–language models spanning image and video understanding. All models are used in their publicly released configurations, without architectural modification. LLaVA-1.5 43 . LLaVA-1.5 is a widely adopted open-source vision–language model that combines a CLIP-based vision encoder with a Vicuna language model. Visual features extracted by the vision encoder are projected into the language embedding space via a linear projection layer, enabling the LLM to process visual tokens directly. Compared to the original LLaVA, LLaVA-1.5 increases the input image resolution from 224\times 224 to 336\times 336 and incorporates additional instruction tuning data, resulting in substantially improved performance across multimodal benchmarks. LLaVA-NeXT (LLaVA-1.6) 44 . LLaVA-NeXT extends LLaVA-1.5 by supporting dynamic high-resolution image inputs. Instead of using a fixed resolution, the model adaptively selects an aspect ratio and partitions high-resolution images into multiple sub-images, each processed independently by the vision encoder. The resulting visual representations are concatenated before being passed to the language model. This design improves performance on tasks requiring fine-grained perception, such as OCR and spatial reasoning, while preserving the underlying architecture. Video-LLaVA 40 . Video-LLaVA extends the LLaVA architecture to video understanding by encoding individual video frames using the same vision encoder and aligning their representations before projection into the language model. Frame-level features are concatenated to form a unified visual token sequence, enabling joint reasoning over spatial and temporal information. After multimodal alignment training, Video-LLaVA supports both image and video-based question answering within a single unified framework. Qwen-VL 4 . Qwen-VL is an open-source vision–language model that pairs an OpenCLIP vision encoder with the Qwen language model. Unlike LLaVA’s linear projection, Qwen-VL employs a vision–language adapter based on cross-attention to map visual features into a fixed-length token sequence. The model is trained in multiple stages to align visual and textual representations, with Qwen-VL-Chat further fine-tuned for conversational multimodal tasks. E.3 Implementation Details Static Cross-Attention Probe (SCAP). SCAP is instantiated as a lightweight replica of the first decoder layer’s attention module and is executed once during the prefill stage. It extracts query-aware text–vision saliency without maintaining key/value caches or modifying the LLM forward pass. As a result, SCAP does not interfere with optimized attention kernels (e.g., FlashAttention) used during subsequent decoding. Training-Free Deployment. ConsensusDrop introduces no additional trainable parameters and is applied entirely at inference time. SCAP parameters are deep-copied from the base model and remain frozen in all experiments, ensuring that performance gains arise solely from improved token selection rather than task-specific adaptation. E.4 Efficiency Analysis Setting All efficiency measurements (Section 5.6 ) are conducted on a single NVIDIA RTX 6000 Ada GPU (48 GB) using CUDA 12.1, PyTorch 2.1.2, and the Hugging Face Transformers library. Unless otherwise stated, we evaluate LLaVA-1.5-7B with SDPA (scaled dot-product attention) enabled and identical generation settings across methods. Time to First Token (TTFT). TTFT measures the prefill latency, defined as the elapsed time from input submission to the generation of the first output token. This metric captures the cost of processing the full multimodal prompt and initializing the KV cache. We report TTFT using GPU-synchronized wall-clock timing, averaged over multiple runs after warm-up. Time per Output Token (TPOT). TPOT measures the steady-state decode latency per generated token. We compute TPOT by generating a fixed-length output sequence with greedy decoding and KV caching enabled, and report the average per-token latency excluding the prefill phase. This metric reflects the decoding cost after the initial prompt has been processed. KV Cache Footprint. The KV cache memory footprint is estimated analytically based on the retained visual token count after pruning. For LLaVA-1.5-7B (LLaMA-7B backbone), the cache size is computed as \text{KV}~(\text{MB})=\frac{2\times L\times H\times S\times D\times B}{2^{20}} , where L=32 is the number of transformer layers, H=32 the number of key-value heads, D=128 the head dimension, S the sequence length (visual and text tokens), and B=2 bytes for FP16 precision. We report the KV cache size immediately after prefill to isolate the impact of visual token reduction. All reported KV cache sizes correspond to the net memory footprint aggregated across all transformer layers. For methods that prune tokens within the LLM, with layer-wise pruning ratios that vary across stages ( 90 ; 73 ), this estimate does not capture potential KV cache memory fragmentation, which can increase effective memory usage in practice. Evaluation Consistency. All methods (FastV 11 , SparseVLM 90 , VisionZip 77 , VisPruner 88 , and ConsensusDrop) are evaluated under identical conditions, including input images, prompts, retention budgets, decoding strategy, and hardware, using their respective official codebase. Baseline results correspond to the full visual token sequence (576 tokens) for LLaVA-1.5-7B. Appendix F Ablation Studies We conduct extensive ablation studies on LLaVA-1.5-7B 46 ; 42 to analyze the contribution of each component in ConsensusDrop. All experiments retain 128 visual tokens (from 576) and evaluate on five benchmarks: MME 19 , POPE 39 , ScienceQA 52 , SEED-Bench 35 , and TextVQA 68 . Unless otherwise noted, we use the default configuration: Convex Fuser with \alpha=0.7 , \tau_{v}=\tau_{c}=1.0 , and EGTM with K=108 selected tokens and M=20 anchor tokens. The default vision-side saliency is the CLS-to-patch attention scores (extracted from the penultimate layer of the vision encoder); the default cross-modal saliency is the aggregate query-to-patch scores extracted from SCAP (Section 4.1 ). F.1 Fusion Weight ( \alpha ) The convex fusion weight \alpha controls the balance between vision-side saliency ( \mathbf{s}^{(v)} ) and cross-modal saliency ( \mathbf{s}^{(c)} ) in the fused score: \mathbf{s}=\alpha\cdot\mathbf{s}^{(v)}+(1-\alpha)\cdot\mathbf{s}^{(c)} . We evaluate \alpha\in\{0.3,0.5,0.7\} , where higher values weight the vision encoder’s saliency more heavily. \alpha MME POPE ScienceQA SEED TextVQA 0.3 1382.3 82.0 68.8 60.9 55.4 0.5 1414.5 83.0 69.3 62.0 55.4 0.7 1451.1 85.1 69.6 62.1 55.1 Table 5 : Effect of fusion weight \alpha . Higher vision weight ( \alpha=0.7 ) yields consistently stronger performance, supporting the hypothesis that vision-encoder saliency provides a reliable base signal that cross-modal attention refines. As shown in Tab. 5 and Fig. 9 , performance improves roughly monotonically as \alpha increases from 0.3 to 0.7. The largest gains appear on POPE (+3.1 F1) and MME (+68.8), both of which require accurate object grounding 39 ; 19 . This suggests that vision-encoder saliency captures spatially coherent regions critical for hallucination-sensitive tasks, while cross-modal attention contributes complementary query-specific refinement. The monotonic improvement validates our design intuition: the vision encoder’s self-attention already encodes semantic groupings (objects, textures, boundaries), making it a reliable “base signal” that cross-modal attention should refine rather than override. (a) Accuracy vs. fusion weight \alpha . (b) Radar plot across all benchmarks. Figure 9 : Effect of fusion weight \alpha across benchmarks. Higher vision weight consistently improves performance, with \alpha=0.7 achieving the best trade-off. F.2 Temperature Scaling ( \tau_{v} , \tau_{c} ) Temperature scaling sharpens or smooths the saliency distributions before fusion. We test \tau_{v},\tau_{c}\in\{0.7,1.3\} , where lower temperatures produce sharper (more peaked) distributions and higher temperatures yield softer distributions. This ablation probes whether ConsensusDrop benefits from amplifying high-confidence tokens (low \tau ) or preserving distributional entropy (high \tau ). \tau_{v} \tau_{c} MME POPE ScienceQA SEED TextVQA 0.7 0.7 1394.2 82.0 69.3 60.6 55.1 0.7 1.3 1418.3 82.3 68.7 61.3 55.3 1.3 0.7 1409.4 82.6 69.4 61.2 55.4 1.3 1.3 1416.9 83.8 69.1 61.9 55.3 Table 6 : Effect of temperature scaling. Performance varies by < 2% across all configurations, indicating robustness to this hyperparameter. Tab. 6 and Fig. 10 demonstrate that ConsensusDrop is largely insensitive to temperature scaling: the maximum variation is 1.8% on POPE and 2.1% on SEED-Bench. This robustness is desirable for practical deployment, as it eliminates the need for dataset-specific tuning. The slight preference for higher temperatures ( \tau_{v}=\tau_{c}=1.3 ) on POPE and SEED-Bench suggests that preserving distributional entropy marginally helps hallucination-sensitive tasks, possibly by avoiding over-concentration on a few dominant tokens. We adopt \tau_{v}=\tau_{c}=1.0 as the default for simplicity, as it lies within the robust operating region. Figure 10 : Temperature ablation heatmaps across all benchmarks. Minimal variance ( < 2%) indicates robustness to temperature scaling, with MME values shown as \times 1/20 for visual comparability. F.3 Recovery Fuser: Student Modality and Recovery Rate The recovery fuser implements a student–teacher correction mechanism (Sec. 4.2 ), where the student modality’s bottom-ranked tokens are replaced by high-ranked tokens from the teacher. We ablate (i) which modality serves as student (visual or cross-modal), and (ii) the recovery rate \gamma\in\{0.1,0.3,0.5\} , which controls what fraction of the student’s lowest-ranked tokens are swapped with the teacher’s highest-ranked tokens. Student \gamma MME POPE ScienceQA SEED TextVQA Visual 0.1 1402.9 83.8 68.7 61.3 55.3 0.3 1404.8 82.9 69.1 60.7 55.2 0.5 1390.7 81.7 69.2 59.9 55.2 Cross 0.1 1320.4 80.4 69.0 58.9 54.1 0.3 1398.2 81.3 68.9 59.3 54.9 0.5 1412.0 82.2 68.6 60.1 55.1 Table 7 : Recovery fuser ablation. Using vision as the student with minimal recovery ( \gamma=0.1 ) yields the best hallucination-sensitive metrics (POPE, SEED-Bench), while cross-as-student requires higher recovery rates to compensate for its weaker base signal. Tab. 7 and Fig. 11 reveal a clear asymmetry: visual-as-student consistently outperforms cross-as-student across all recovery rates, with the gap most pronounced at low recovery rates (POPE: 83.8 vs. 80.4 at \gamma=0.1 ). This aligns with our hypothesis that the vision encoder provides a stable base signal, and cross-modal attention should act as a corrective teacher rather than the primary selector. Interestingly, higher recovery rates ( \gamma=0.5 ) degrade POPE and SEED-Bench when visual is the student, suggesting that excessive correction disrupts the encoder’s coherent spatial signal. Conversely, cross-as-student improves with higher recovery rates, indicating that cross-modal saliency alone is insufficient and requires substantial correction from the vision teacher. This asymmetry provides mechanistic evidence for our design choice of vision-weighted fusion. Figure 11 : Recovery fuser: POPE F1 vs. recovery rate \gamma . Visual-as-student (green) consistently outperforms cross-as-student (red), with the gap widening at lower recovery rates. F.4 Anchor Tokens (EGTM) EGTM partitions the token budget into K directly selected tokens and M merged anchor tokens, such that K+M=128 . We sweep M\in\{0,10,20,30,40,50\} (correspondingly, K\in\{128,118,108,98,88,78\} ) to study the trade-off between selection fidelity and contextual preservation through merging. K M MME POPE ScienceQA SEED TextVQA 128 0 1410.2 81.3 69.4 61.5 55.0 118 10 1415.8 82.8 69.6 61.8 55.1 108 20 1441.2 84.9 69.5 63.2 57.0 98 30 1430.0 83.6 69.1 62.0 55.0 88 40 1420.6 84.3 69.0 62.2 55.4 78 50 1418.7 84.7 69.3 62.1 55.5 Table 8 : EGTM anchor token ablation. M=20 achieves the optimal balance, improving POPE by +3.6 F1 and TextVQA by +2.0% over pure selection ( M=0 ). Tab. 8 and Fig. 12 demonstrate that EGTM provides consistent gains over pure top- K selection. The sweet spot occurs at M=20 : compared to M=0 (no merging), this configuration improves POPE by +3.6 F1, SEED-Bench by +1.7%, and TextVQA by +2.0%. Performance degrades slightly for M>20 , as excessive merging dilutes the information preserved in anchor tokens—the merged representation becomes an over-smoothed average rather than a focused summary. This validates our EGTM design: a modest merge budget recovers contextual information from discarded tokens (background, peripheral objects) without sacrificing the precision of top- K selection for salient foreground regions. (a) Accuracy vs. anchor count M . (b) Improvement over M=0 baseline. Figure 12 : EGTM anchor token trade-off. (a) All benchmarks peak near M=20 . (b) Merging provides up to +3.6% improvement over pure selection, validating the value of contextual preservation. F.5 Score Type We ablate the choice of saliency extraction for both modalities: (i) vision score type \in {tokens, class}, where “tokens” computes mean patch-to-patch attention and “class” uses CLS-to-patch attention from the vision encoder; and (ii) cross-modal score type \in {all, last, max}, where “all” averages attention from all text tokens to each visual token, “last” uses only the final text token’s attention, and “max” takes the element-wise maximum across text tokens. Vision Cross MME POPE ScienceQA SEED TextVQA Avg. tokens all 1431.1 83.9 68.8 61.5 55.5 67.4 class all 1467.3 83.7 69.1 63.4 56.2 68.1 tokens last 1443.5 82.3 68.0 61.8 55.8 67.0 class last 1420.5 83.7 69.2 61.9 55.2 67.5 tokens max 1436.5 83.6 69.7 61.0 55.6 67.5 class max 1424.1 83.5 69.1 62.0 55.2 67.5 Table 9 : Score type ablation. The “class+all” configuration achieves the highest average accuracy (+0.7% over second-best) with the strongest MME, SEED-Bench, and TextVQA scores. Tab. 9 and Fig. 13 show that the “class+all” configuration achieves the best overall performance. CLS-to-patch attention provides a globally-informed saliency signal that outperforms local patch-to-patch attention, particularly on MME (+36.2 over tokens+all) and SEED-Bench (+1.9%). This is expected: the CLS token aggregates global context during vision encoding, making its attention pattern a better proxy for “image-level importance” than local patch interactions 17 . Averaging cross-modal attention across all text tokens (“all”) outperforms “last” and “max”, likely because it captures the aggregate query intent. Using only the last token (“last”) may overweight positional recency, while “max” can be sensitive to spurious attention spikes on individual text tokens. Figure 13 : Score type ablation heatmap. The “class+all” configuration (highlighted) achieves the strongest performance across most benchmarks, with MME values shown as \times 1/20 . F.6 Summary Figure 14 : Grouped bar comparison with LLaVA baseline (dashed line). All best configurations approach the LLaVA baseline (576 tokens) while using only 128 tokens (78% reduction). Score type (class+all) and anchor tokens ( M=20 ) achieve the highest absolute scores. Fig. 15 and Fig. 14 summarize the best configuration from each ablation study. Across all experiments, the default ConsensusDrop configuration ( \alpha=0.7 , \tau_{v}=\tau_{c}=1.0 , visual student with \gamma=0.1 , K=108 , M=20 , class+all scoring) consistently achieves strong performance, retaining over 95% of baseline accuracy on hallucination-sensitive benchmarks while reducing visual tokens by 78%. Key findings include: Figure 15 : Radar view showing tight clustering across all optimal configurations. • Vision-weighted fusion ( \alpha=0.7 ) outperforms balanced or cross-weighted fusion by +68.8 MME and +3.1 POPE F1, indicating that encoder saliency provides a reliable base signal that cross-modal attention should refine. • Temperature scaling has minimal impact ( < 2% variation across all benchmarks), demonstrating robustness and eliminating the need for dataset-specific tuning. • Visual-as-student in the recovery fuser outperforms cross-as-student by up to +3.4 POPE F1 at low recovery rates, supporting the asymmetric role of vision (base) and cross-modal (refinement). • EGTM with M=20 anchors provides +3.6 POPE F1 and +2.0 TextVQA over pure top- K selection, validating the value of preserving contextual information through token merging. • CLS-to-patch attention (“class”) combined with full-text averaging (“all”) yields the best score extraction strategy, leveraging global context from both modalities. Table 10 : Efficiency comparison on LLaVA-1.5-13B (576 visual tokens baseline). Larger models exhibit proportionally greater efficiency gains from token pruning. \dagger SparseVLM uses progressive layer-wise pruning; see Table 11 caption. Method Reduction # Tokens TTFT (ms) Speedup FLOPs (T) KV Cache (MB) GPU Mem (GB) Gen. Time (ms) Baseline 0% 576 156.5 1.00 \times 16.56 489 26.70 824.6 25% Token Reduction ConsensusDrop (Ours) 25% 432 138.8 1.13 \times 12.84 377 26.66 800.4 VisionZip 25% 432 135.1 1.16 \times 12.84 377 26.47 792.5 VisPruner 25% 432 112.9 1.39 \times 13.24 377 25.87 762.7 SparseVLM † 25% 432 92.0 1.70 \times 10.61 334 32.40 960.7 50% Token Reduction ConsensusDrop (Ours) 50% 288 106.7 1.47 \times 9.13 264 26.42 758.9 VisionZip 50% 288 103.5 1.51 \times 9.13 264 26.22 752.6 VisPruner 50% 288 84.7 1.85 \times 9.52 264 25.64 700.9 SparseVLM † 50% 288 93.1 1.68 \times 6.75 234 32.39 962.7 75% Token Reduction ConsensusDrop (Ours) 75% 144 82.5 1.90 \times 5.45 152 26.40 729.1 VisionZip 75% 144 80.6 1.94 \times 5.45 152 26.20 724.4 VisPruner 75% 144 70.5 2.22 \times 5.83 152 25.41 639.0 SparseVLM † 75% 144 83.9 1.87 \times 4.19 155 32.30 952.8 90% Token Reduction ConsensusDrop (Ours) 90% 57 66.5 2.35 \times 3.23 84 26.40 710.6 VisionZip 90% 57 62.4 2.51 \times 3.23 84 26.20 706.6 VisPruner 90% 57 64.4 2.43 \times 3.60 84 25.37 601.7 SparseVLM † 90% 57 77.3 2.02 \times 2.99 109 32.28 948.3 95% Token Reduction ConsensusDrop (Ours) 95% 28 64.3 2.43 \times 2.49 61 26.40 706.1 VisionZip 95% 28 60.2 2.60 \times 2.49 61 26.20 702.2 VisPruner 95% 28 64.5 2.43 \times 2.86 61 25.37 589.2 SparseVLM † 95% 28 75.7 2.07 \times 2.61 93 32.31 944.4 Appendix G Additional Efficiency Analysis This section provides comprehensive efficiency benchmarks comparing ConsensusDrop against VisionZip 77 , VisPruner 88 , and SparseVLM 90 across LLaVA-1.5-7B and LLaVA-1.5-13B models. The experimental settings are identical to those described in Section E.4 . All methods are evaluated under matched token retention budgets. G.1 Efficiency Metrics Summary We report metrics as defined in Section E.4 , along with Peak GPU Memory , the maximum memory consumption during inference. For each metric, we compute both absolute values and relative improvements over the baseline (576 visual tokens). G.2 LLaVA-1.5-7B Results Table 11 presents detailed efficiency metrics for LLaVA-1.5-7B across five token reduction ratios (25%–95%). At moderate reduction levels (50%), ConsensusDrop achieves a 1.35 \times TTFT speedup with 46% KV cache reduction , closely matching VisionZip 77 while outperforming SparseVLM in generation latency. At aggressive reduction (90%), ConsensusDrop delivers 1.71 \times TTFT speedup and 83% KV cache reduction . Notably, while VisPruner 77 exhibits marginally faster TTFT at some operating points, this advantage comes at the cost of its iterative pruning loop, which incurs additional computational overhead reflected in higher FLOPs. SparseVLM’s 90 progressive layer-wise pruning strategy results in significantly higher peak GPU memory (+3.7 GB over ConsensusDrop at 50% reduction) due to its multi-stage KV cache management. Table 11 : Efficiency comparison on LLaVA-1.5-7B (576 visual tokens baseline). ConsensusDrop achieves competitive TTFT speedups while maintaining lower memory overhead than SparseVLM. \dagger SparseVLM uses progressive layer-wise pruning; token count shown is the target budget (actual final-layer tokens are fewer due to its multi-stage schedule). Method Reduction # Tokens TTFT (ms) Speedup FLOPs (T) KV Cache (MB) GPU Mem (GB) Gen. Time (ms) Baseline 0% 576 90.0 1.00 \times 8.47 313 14.50 471.6 25% Token Reduction ConsensusDrop (Ours) 25% 432 82.2 1.09 \times 6.57 241 14.48 458.0 VisionZip 25% 432 82.5 1.09 \times 6.57 241 14.48 464.7 VisPruner 25% 432 57.4 1.57 \times 6.95 241 13.78 436.2 SparseVLM † 25% 432 60.4 1.49 \times 5.61 212 18.15 626.7 50% Token Reduction ConsensusDrop (Ours) 50% 288 66.9 1.35 \times 4.67 169 14.48 446.7 VisionZip 50% 288 65.2 1.38 \times 4.67 169 14.34 440.7 VisPruner 50% 288 50.8 1.77 \times 5.05 169 13.60 400.9 SparseVLM † 50% 288 61.1 1.47 \times 3.63 148 18.15 625.9 75% Token Reduction ConsensusDrop (Ours) 75% 144 52.6 1.71 \times 2.78 97 14.48 430.7 VisionZip 75% 144 50.9 1.77 \times 2.78 97 14.36 437.1 VisPruner 75% 144 46.8 1.92 \times 3.16 97 13.52 365.5 SparseVLM † 75% 144 62.6 1.44 \times 2.32 96 18.15 614.1 90% Token Reduction ConsensusDrop (Ours) 90% 57 47.5 1.89 \times 1.64 54 14.44 430.5 VisionZip 90% 57 45.9 1.96 \times 1.64 54 14.36 434.3 VisPruner 90% 57 48.7 1.85 \times 2.02 54 13.52 344.1 SparseVLM † 90% 57 63.2 1.42 \times 1.62 56 18.15 619.6 95% Token Reduction ConsensusDrop (Ours) 95% 28 46.0 1.96 \times 1.27 39 14.48 426.8 VisionZip 95% 28 42.7 2.11 \times 1.27 39 14.36 434.4 VisPruner 95% 28 46.3 1.94 \times 1.64 39 13.52 337.0 SparseVLM † 95% 28 45.2 1.99 \times 1.51 41 18.15 613.8 Figure 16 : Comprehensive efficiency metrics comparison across token reduction ratios. Each subplot shows a different metric: (a) TTFT latency, (b) TTFT speedup over baseline, (c) KV cache memory footprint, (d) computational FLOPs, (e) peak GPU memory, and (f) KV cache reduction percentage. Solid lines indicate LLaVA-1.5-7B; dashed lines indicate LLaVA-1.5-13B. ConsensusDrop (red) achieves efficiency profiles comparable to VisionZip while avoiding the memory overhead of SparseVLM. G.3 LLaVA-1.5-13B Results Table 10 extends the analysis to the larger LLaVA-1.5-13B model. The efficiency gains from visual token pruning scale consistently with model size. At 50% reduction, ConsensusDrop achieves 1.56 \times TTFT speedup and 46% KV cache reduction . The larger model amplifies the memory advantages of ConsensusDrop over SparseVLM, with the latter requiring +6.0 GB additional GPU memory at the 50% reduction level. G.4 Efficiency vs. Accuracy Trade-off The key insight from our efficiency analysis is that ConsensusDrop achieves Pareto-optimal accuracy–efficiency trade-offs . While VisPruner demonstrates faster TTFT at certain operating points, this comes at the cost of (i) higher computational FLOPs due to its iterative pruning loop, and (ii) reduced accuracy (and inference performance) at aggressive reduction ratios (Section 5 ). Similarly, while SparseVLM’s progressive pruning can achieve competitive TTFT, it requires substantially more GPU memory and exhibits slower end-to-end generation times. ConsensusDrop’s efficiency profile is characterized by: • Consistent TTFT speedups (1.35–2.0 \times ) that scale with token reduction ratio • Linear KV cache reduction directly proportional to token reduction • Minimal memory overhead compared to the baseline, unlike SparseVLM’s +3–6 GB overhead • Stable generation latency that does not degrade with aggressive pruning Figure 17 visualizes these trends across both model sizes, demonstrating that ConsensusDrop maintains favorable efficiency characteristics while preserving accuracy (Section 5 ). Figure 17 : Efficiency comparison across token reduction ratios. (a, b) TTFT decreases consistently with token reduction for all methods. (c, d) KV cache memory scales linearly with retained tokens. ConsensusDrop (green/red/blue; coinciding) achieves competitive efficiency while avoiding the memory overhead of SparseVLM (purple) and the computational overhead of VisPruner. Figure 18 provides a grouped comparison of TTFT speedups across methods at fixed reduction levels. At moderate reduction (50%), all methods achieve meaningful speedups (1.35–1.77 \times for 7B), with differences becoming more pronounced at higher reduction ratios. Importantly, ConsensusDrop’s speedup profile closely tracks that of VisionZip, confirming that our multimodal consensus mechanism introduces negligible overhead relative to vision-only pruning. Figure 18 : TTFT speedup comparison at fixed reduction levels. ConsensusDrop achieves speedups comparable to VisionZip while providing superior accuracy through multimodal consensus. SparseVLM’s speedups plateau due to its progressive pruning overhead. Appendix H Qualitative Visualizations We present qualitative visualizations demonstrating ConsensusDrop’s token selection mechanism (examples from VQAv2 21 test-dev2015 split, using LLaVA-1.5-7B as the base VLM). Our consensus fusion approach combines vision-based saliency scores with cross-modal attention to identify tokens that are both visually salient and semantically relevant to the question. H.1 Token Selection Comparison Figure 19 and 20 shows side-by-side comparisons of token selection strategies: Vision-Only selects tokens based on visual saliency from the CLIP encoder, Cross-Modal Only uses attention scores from text-to-image attention, and Consensus combines both through our convex fusion ( \alpha=0.7 ). The visualizations reveal that vision-only selection often focuses on globally salient regions (e.g., high-contrast edges, textures) while missing question-relevant details. Cross-modal attention, affected by positional biases (RoPE), tends to over-weight late positions (bottom image patches). Our consensus fusion balances these complementary signals, retaining tokens that are both visually informative and semantically aligned with the question. H.2 High-Disagreement Examples We analyze examples with the highest modality disagreement—cases where vision-only and cross-modal attention select substantially different token sets (Figure 21 to 30 ). These examples best demonstrate the value of consensus fusion. At higher compression rates, careful token selection becomes even more critical. These examples demonstrate effectiveness of ConsensusDrop under aggressive pruning. H.3 Key Observations • Complementary signals: Vision-based scores capture global saliency, while cross-modal attention identifies question-relevant regions. Neither alone is sufficient. • Positional bias in cross-modal: Cross-modal attention exhibits positional biases due to RoPE, sometimes over-weighting tokens at sequence boundaries rather than semantically relevant ones, a known issue 88 in VLMs with a language model backbone using RoPE embeddings. • Consensus benefit: Our convex fusion ( \alpha=0.7 ) successfully combines both signals, retaining tokens that are both visually salient and semantically aligned. • Robustness across K: The consensus mechanism remains effective across different retention ratios, from moderate (K=288) to aggressive (K=128) compression. Figure 19 : Token selection comparison at K=288 (50% reduction). Each panel shows: Original image, Vision-Only selection (green), Cross-Modal Only selection (blue), and Consensus fusion (magenta). Questions and model answers are shown below each comparison. Figure 20 : Token selection comparison at K=128 (78% reduction). At higher compression, the disagreement between vision and cross-modal signals becomes more pronounced, making consensus fusion even more critical. (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 21 : Q: “What is the dog trying to catch?” A: “The dog is trying to catch a frisbee….” (Disagreement: 61.1%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 22 : Q: “Is he fixing his tie?” A: “Yes, the man in the image is fixing his tie, which is a common activity before a…” (Disagreement: 54.9%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 23 : Q: “What is on this man’s wrist?” A: “The man is wearing a watch on his wrist….” (Disagreement: 54.9%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 24 : Q: “Is the guy already done surfing?” A: “Yes, the guy is already done surfing and is walking out of the water with his su…” (Disagreement: 54.2%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 25 : Q: “Where are the wheels?” A: “The wheels are on the table….” (Disagreement: 54.2%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 26 : Q: “How many people are visible?” A: “There is only one person visible in the image….” (Disagreement: 93.0%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 27 : Q: “What season is it?” A: “It is winter….” (Disagreement: 93.0%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 28 : Q: “Is everyone wearing a hat?” A: “Yes, everyone in the image is wearing a hat….” (Disagreement: 90.6%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 29 : Q: “What color is the couch?” A: “The couch is brown….” (Disagreement: 89.1%) (a) Attention heatmaps (b) Token set overlap (c) Minimal comparison (d) Across retention ratios Figure 30 : Q: “What are they doing?” A: “The people in the image are skiing and snowboarding on a snow-covered slope….” (Disagreement: 88.3%) Appendix I Limitations and Discussion ConsensusDrop relies on the availability of meaningful cross-modal attention signals; for prompts with extremely weak or ambiguous visual grounding, the benefit of consensus-based fusion may diminish. While SCAP enables efficient pre-LLM extraction of cross-modal cues, it remains a lightweight approximation of full LLM cross-attention and may not fully capture higher-layer semantic interactions. Our approach focuses on visual token reduction and does not address redundancy in text tokens or inter-layer token dynamics within the LLM. Finally, although encoder-guided token merging preserves semantic context in practice, aggressive compression may still lead to loss of fine-grained spatial details in highly localized visual reasoning tasks. Appendix J Reproducibility Our implementation builds upon the official open-source codebases of LLaVA 46 , Video-LLaVA 40 , and Qwen-VL 5 , with all model variants, preprocessing pipelines, and evaluation protocols preserved unless explicitly stated. We will release the full ConsensusDrop codebase, including inference scripts, configuration files, and evaluation utilities, to enable reproduction of all reported results. The main paper and appendix provide exhaustive details on experimental settings, token retention budgets, efficiency measurement, and ablation configurations to ensure transparency and reproducibility. Appendix K Use of Large Language Models Large Language Models were used as auxiliary tools to assist with code refactoring, experiment orchestration across ablation settings, and minor editing for clarity and grammar after the core methods, implementations, and experiments were developed by the authors. All algorithmic design, experimental decisions, result interpretation, and final writing remain the sole responsibility of the authors.