Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles

Zacharie BUGAUD · 2026

Read Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles by Zacharie BUGAUD (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-17111.txt.

Introduction

Bugaud exposes a family bias hidden inside vision language model ensembles. Ensembles mix models from many providers to raise accuracy, but models from one family share correlated errors. Across 17 models from 8 families, he finds family correlation cuts the effective voter count to about 3. On a small misleading tier of questions, correlated majorities drive accuracy to zero even when the best model is right. He proposes three family aware fixes. The strongest, learned candidate scoring, gains on all three benchmarks and never degrades any of them.

Catalog record

CREATORZacharie BUGAUD
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42761
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.17111

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Hidden Clones: Exposing and Fixing Family Bias in Vision-Language Model Ensembles Zacharie BUGAUD Abstract Ensembling Vision-Language Models (VLMs) from different providers maximizes benchmark accuracy, yet models from the same architectural family share correlated errors that standard voting ignores. We study this structure across 17 VLMs from 8 families on VQAv2, TextVQA, and GQA. Family-correlated errors reduce effective ensemble dimensionality to 2.5–3.6 independent voters and create a Misleading tier (1.5–6.5% of questions) where correlated majority errors destroy accuracy to 0% despite the best model being correct. We propose three family-aware methods. Hierarchical Family Voting (HFV) aggregates within families before voting across them, recovering +18–26 pp on the Misleading tier. QualRCCV , a training-free method weighting models by calibration, family quality, and inverse family size, is the first to beat calibrated voting on all three benchmarks ( p < 0.05 p{<}0.05 ). Learned Candidate Scoring (LCS) trains a cross-validated classifier to re-rank candidate answers using support breadth, family diversity, and model quality, achieving the largest gains: + 0.68 % +0.68\% VQAv2, + 0.61 % +0.61\% TextVQA, + 2.45 % +2.45\% GQA—all significant—and is the only learned method that never degrades any benchmark. On VQAv2 test-standard (EvalAI), LCS reaches 87.83% with 12 models, confirming generalization. 1 Introduction Combining predictions from multiple models is the default strategy for maximizing accuracy in visual question answering (VQA) competitions jiang2020defense ; yu2019deep and across machine learning more broadly. Condorcet’s jury theorem condorcet1785 provides the theoretical motivation: majority voting improves with more voters, provided each voter is better than random and errors are independent. Conventional wisdom further holds that diverse ensembles—combining different architectures—outperform homogeneous ones dietterich2000ensemble . In practice, state-of-the-art VLM ensembles are constructed from models belonging to a small number of architectural families —e.g. Qwen2.5-VL, Qwen3-VL, InternVL, Molmo, Phi-4, LLaVA-OneVision, Pixtral, Idefics3—where models within a family share training data, architecture, and pre-training methodology. This creates a hidden structure that standard ensemble methods ignore: within-family errors are strongly correlated , violating the independence assumption that makes voting powerful. We present the first multi-benchmark study of this family structure, spanning 17 VLMs from 8 families across VQAv2 ( N{=}20{,}001 ), TextVQA ( N{=}5{,}000 ), and GQA ( N{=}12{,}578 ). Our contributions are: 1. A multi-benchmark analysis of family-correlated errors revealing that eigenvalue structure reduces 17 models to only 2–4 effective voters, and a difficulty taxonomy identifying a Misleading tier (1.5–6.5% of questions) where calibrated voting collapses to 0% despite the best model being correct (Section 4 ). 2. Hierarchical Family Voting (HFV) , a training-free method that aggregates within families first and then across them, recovering the Misleading tier by +18–26 pp . HFV-sharp , with cross-validation for \alpha , achieves 87.19% on VQAv2 ( +0.49\% , p{<}0.0001 ) and 64.27% on GQA ( +0.25\% , p{=}0.087 ), remaining entirely training-free (Section 5 ). 3. Quality-weighted Redundancy-Corrected Calibrated Voting (QualRCCV) , a training-free single-level vote that weights each model by w_{m}\cdot q_{f}^{\gamma}/|F(m)|^{\rho} where q_{f} is the family’s best-member accuracy. QualRCCV is the first method to beat calibrated voting on all three benchmarks : +0.17\% VQAv2 ( p{=}0.003 ), +0.21\% TextVQA ( p{=}0.034 ), +0.31\% GQA ( p{=}0.003 ), remaining training-free with two hyperparameters (Section 5.4 ). 4. Learned Candidate Scoring (LCS) , a cross-validated method that scores individual candidate answers based on per-answer features (support breadth, family diversity, supporter quality). LCS achieves the largest gains of any method: +0.68\% on VQAv2 ( p{<}0.0001 ) and +2.45\% on GQA ( p{<}0.0001 ), while remaining positive on TextVQA ( +0.61\% , p{<}0.0001 ). LCS is the only learned method that never degrades any benchmark (Section 6 ). 2 Related Work Ensemble theory and diversity. Condorcet’s jury theorem condorcet1785 shows majority voting improves with more independent, better-than-random voters. Extensions to correlated voters ladha1992condorcet ; berg1993random ; boland1989modelling predict ensemble degradation when errors are positively correlated. The bias–variance–covariance decomposition ueda1996generalization ; brown2005diversity formalizes how ensemble error depends on both individual accuracy and pairwise diversity. Kuncheva & Whitaker kuncheva2003measures survey diversity measures and show that no single measure reliably predicts ensemble accuracy. Our work contributes an empirical diversity analysis for VLM ensembles, revealing that architectural family membership is the dominant source of correlation structure. LLM and VLM ensembles. LLM-Blender jiang2023llmblender trains a ranking model to select the best response from multiple LLMs. Mixture-of-Agents wang2024mixture iteratively refines outputs by passing responses through multiple LLMs. RouteLLM ong2024routellm and FrugalGPT chen2023frugalgpt train routers or cascades to optimize cost–quality trade-offs. More-Agents-Is-All-You-Need li2024more shows scaling the number of LLM agents improves performance on reasoning tasks through majority voting. Wang et al. wang2023selfconsistency introduce self-consistency decoding, sampling multiple reasoning paths from a single model and voting. In contrast to methods requiring training data or iterative generation, our HFV method is training-free and operates on answer-level outputs from heterogeneous models. Structured and hierarchical aggregation. Hierarchical voting appears in social choice theory (e.g., electoral colleges condorcet1785 ) and in ensemble learning via stacking wolpert1992stacked and mixture of experts jacobs1991adaptive ; shazeer2017outrageously . Nested cross-validation and meta-learning approaches van2007super aggregate base learners in stages. To our knowledge, we are the first to apply hierarchical architecture-family-level aggregation to VLM ensembles and to analyze when it helps versus hurts. VQA benchmarks and evaluation. VQAv2 goyal2017vqav2 introduced balanced image pairs to reduce language bias; TextVQA singh2019textvqa requires OCR reasoning; GQA hudson2019gqa tests compositional reasoning via scene graphs. Prior ensemble work on VQA focuses on homogeneous ensembles of task-specific models jiang2020defense ; yu2019deep ; we study heterogeneous ensembles of general-purpose VLMs. 3 Experimental Setup Models. We assemble 17 VLMs from 8 architectural families (Table 1 ): 5 Qwen2.5-VL variants (7B fine-tuned on VQAv2, two 7B LoRA variants, 32B and 72B zero-shot), 2 Qwen3-VL variants (8B and 32B zero-shot), 2 InternVL variants (InternVL2-8B and InternVL3-8B, both zero-shot), 2 Molmo2-8B variants (one with prompt engineering, one raw), Phi-4-multimodal (14B zero-shot), 2 LLaVA variants (OneVision-7B and LLaVA-NeXT-Mistral-7B zero-shot), Pixtral-12B zero-shot, and 2 Idefics variants (Idefics3-8B and SmolVLM-2B zero-shot). All inference uses vLLM v0.11 or HuggingFace Transformers. Benchmarks. We evaluate on three VQA benchmarks: • VQAv2 goyal2017vqav2 : minival split, N{=}20{,}001 questions evenly split across yes/no, number, and other types (33.3% each). Soft accuracy with 10 annotators. • TextVQA singh2019textvqa : val split, N{=}5{,}000 questions requiring OCR and text reasoning. Soft accuracy with 10 annotators. • GQA hudson2019gqa : testdev split, N{=}12{,}578 questions from scene-graph-based compositional reasoning. Exact-match accuracy. Aggregation baselines. We compare: majority voting (unweighted), calibrated voting (per-model log-odds weights based on overall accuracy), deduplication (best model per family, then calibrated vote), correlation-aware weighting (inverse-agreement weights), and the per-question oracle (selecting the best answer for each question). Statistical testing. All confidence intervals are 95% bootstrap CIs (2,000 resamples). Significance of HFV vs. calibrated voting is assessed via a paired bootstrap test: we resample questions with replacement and compute the fraction of resamples where calibrated voting outperforms HFV. We report this as a one-sided p -value. Table 1: Model inventory and individual accuracy across three benchmarks. Family dominance varies: Molmo leads on VQAv2, while Qwen2.5-VL LoRA variants lead on TextVQA; LLaVA-NeXT leads on GQA. Models fine-tuned on VQAv2 (fullft) transfer poorly to TextVQA (67%). InternVL3 and Phi-4 collapse below 50% on TextVQA. All 17 models evaluated on all benchmarks. Model Family Size VQAv2 TextVQA GQA molmo2raw Molmo 8B 86.3 77.9 59.0

Read the complete text