Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models Journal: Knowledge-Based Systems
Zhipeng Ye, Jiaqi Huang, Feng Jiang, Qiufeng Wang, Yikang Duan, Dawei Wang, Xihang Zhou, Qian Qiao · 2026
Read Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models Journal: Knowledge-Based Systems by Zhipeng Ye, Jiaqi Huang, Feng Jiang, Qiufeng Wang, Yikang Duan, Dawei Wang, Xihang Zhou, Qian Qiao (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24799.txt.
Introduction
Zhipeng Ye and co-authors diagnose Performance Collapse in Long Sequence Recognition: multimodal models degrade on image classification as labels multiply. Analysis traces the collapse to rising information entropy clashing with attention dilution and decay. That clash erodes the signal-to-noise ratio over long prompts. Their Divide-and-Conquer Inference splits global classification into simpler local subproblems, with dynamic pruning shrinking the search space. On ImageNet-1K and ImageNet-21K it lifts accuracy, letting lightweight open-source models rival frontier closed-source giants with no extra training.
Catalog record
| CREATOR | Zhipeng Ye, Jiaqi Huang, Feng Jiang, Qiufeng Wang, Yikang Duan, Dawei Wang, Xihang Zhou, Qian Qiao |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60029 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24799
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Divide-and-Conquer Inference for Large-Scale Visual Recognition with Multimodal Large Language Models Journal: Knowledge-Based Systems Zhipeng Ye, Jiaqi Huang, Feng Jiang, Qiufeng Wang, Yikang Duan, Dawei Wang, Xihang Zhou, Qian Qiao Abstract Multimodal Large Language Models (MLLMs) have demonstrated strong capabilities across a wide range of vision-language tasks. However, when applied to large-scale image classification, their performance degrades significantly as the label space expands—a phenomenon we define as Performance Collapse in Long Sequence Recognition. Through an information-theoretic analysis, we reveal that this collapse stems from a fundamental conflict between the escalating information entropy and the prominent attention dilution and decay within attention mechanisms, which impairs the model’s ability to maintain a sufficient signal-to-noise ratio when processing extremely long prompts. To mitigate this, we propose Divide-and-Conquer Inference (DCI), a novel test-time scaling strategy for visual recognition with MLLMs. DCI recursively decomposes complex global classification tasks into multiple simpler, localized sub-problems and employs a dynamic pruning mechanism to compress the search space. This method effectively improves the local signal-to-noise ratio and model accuracy by mitigating the inherent weight dilution issues in long-sequence inference. Moreover, while traditional self-attention incurs a prohibitive quadratic computational complexity, DCI achieves more favorable scaling behavior and substantially accelerates inference in large-scale classification scenarios. Extensive experiments on benchmarks such as ImageNet-1K and ImageNet-21K demonstrate that DCI consistently improves classification accuracy. This enables lightweight open-source models to rival or even surpass frontier closed-source giants without any additional training or fine-tuning. As a model-agnostic, plug-and-play paradigm, DCI offers an efficient approach for scaling the inferential precision of MLLMs in large-scale scenarios. To facilitate reproducibility and further research, the source code is publicly available at: https://github.com/FourierAI/DCI . † † highlights: We identify performance degradation in long-sequence recognition. A Divide-and-Conquer Inference framework is proposed for MLLMs. DCI improves classification accuracy without additional training. DCI reduces inference overhead in large-scale recognition tasks. Extensive experiments validate scalability across diverse MLLMs. 1 Introduction In recent years, Multimodal Large Language Models (MLLMs) [ 34 ] have emerged as a cornerstone of Artificial Intelligence research. Governed by the Scaling Law [ 16 ] , models such as GPT-4V [ 1 ] , Qwen3-VL [ 30 ] , and DeepSeek-VL [ 20 ] have achieved state-of-the-art performance by scaling up datasets and parameter counts. Beyond synergistic multimodal understanding, these models exhibit profound emergent abilities, including complex reasoning, in-context learning, and self-reflection. Consequently, MLLMs are increasingly viewed as a viable path toward Artificial General Intelligence (AGI) [ 10 ] , sparking a surge of interest in adapting them for traditional Computer Vision (CV) tasks. Prior research indicates that distilling open-world knowledge from MLLMs into vision models can substantially boost performance. For instance, LLM2CLIP [ 29 ] replaces the CLIP [ 22 ] text encoder with a pre-trained Large language Model (LLM) and employs lightweight fine-tuning to achieve state-of-the-art (SOTA) results. LaCLIP [ 8 ] leverages in-context learning to rewrite image captions into semantically diverse versions, enhancing robustness through “text-side random augmentation.” Similarly, IDEA [ 33 ] utilizes MLLMs to generate fine-grained descriptions, using an Adapter to heighten feature perception in few-shot scenarios. Moreover, LLaVA-OneVision [ 18 ] unifies diverse vision tasks into a Visual Question Answering (VQA) framework by aligning a frozen SigLIP encoder with the Qwen2 [ 31 ] language model, significantly improving model versatility. Despite their success, directly applying MLLMs to large-scale image classification remains challenging. While a prevalent view [ 35 ] attributes this to a lack of domain-specific training data, we argue that the primary bottleneck arises from the growing size of the label space itself. To verify this, we conduct extensive benchmarks on ImageNet-1K using mainstream MLLMs (e.g., LlaMA [ 11 ] , Gemma [ 24 ] , and Qwen [ 30 ] ). By sampling label subsets of sizes \mathcal{S}\in\{10,20,100,200,\allowbreak 500,1000\} , we observe a consistent trend (see Fig. 1 ): classification accuracy degrades precipitously as the label set size increases , regardless of model architecture or scale. We refer to this phenomenon as Performance Collapse in Long Sequence Recognition (PC-LSR). Notably, LlaMA 3.2-11B-Vision maintains 88.60% accuracy on 20 classes but suffers a catastrophic performance collapse as the set grows to 200, eventually dropping to a mere 0.53% accuracy on the full 1000-class set. Figure 1: LLM performance on ImageNet-1K across varying numbers of candidate classes. The x-axis represents the specific number of candidate classes presented to the model in a closed-set classification task, sampled at intervals of 10, 20, 100, 200, 500, and 1,000. The y-axis shows the resulting score. Each line represents the performance trajectory of a specific MLLM as the candidate space expands. In this paper, we provide an information-theoretic explanation for this phenomenon: (1) Uncertainty Scaling , where increasing label set size raises information entropy and thus decision complexity; and (2) Attention Dilution , where long candidate lists reduce the Signal-to-Noise Ratio (SNR) of attention, impairing the model’s ability to identify the correct label amid excessive textual noise. To mitigate these issues, we propose Divide-and-Conquer Inference (DCI) , a simple yet potent strategy tailored for image classification. Inspired by the divide-and-conquer paradigm, DCI partitions a massive category list into multiple manageable subsets for local inference. By recursively aggregating results, DCI transforms a complex global search into a series of simplified local decisions. This “coarse-to-fine” refinement acts as a dynamic attention mask, effectively reducing the influence of irrelevant categories and preserving the SNR. We evaluate DCI across multiple benchmarks (ImageNet-21K [ 7 ] , ImageNet-1K [ 7 ] , CIFAR-100 [ 17 ] , CUB-200 [ 28 ] , Food-101 [ 4 ] ) using various MLLMs. Experimental results show that DCI consistently boosts performance; notably, an open-source Qwen3-VL-8B [ 30 ] equipped with DCI can achieve competitive performance compared to trillion-parameter scale models, such as GPT-4 [ 1 ] and Qwen3-VL-PLUS [ 30 ] . Moreover, in large-scale image classification scenarios, while standard flat inference suffers from a prohibitive \mathcal{O}(N^{2}) computational complexity due to global context bottlenecks, DCI substantially reduces the computational overhead, delivering significantly faster inference speeds. Our contributions are summarized as follows: 1. Systematic Analysis : To better understand this behavior, we conduct a systematic empirical study of the performance collapse of MLLMs under large-scale label sets and provide a information-theoretic analysis of its underlying causes. 2. Inference Strategy : We introduce Divide-and-Conquer Inference (DCI), a novel paradigm that recursively decomposes complex classification tasks into parallel, simple sub-tasks, effectively addressing noise interference in long-context attention. 3. Empirical Validation : Extensive experiments show that DCI is a training-free, model-agnostic, and plug-and-play framework. In large-scale image classification scenarios, DCI simultaneously delivers higher recognition accuracy and reduced computational latency, allowing lightweight open-source models to rival frontier closed-source giants. 2 Related Work 2.1 Test-Time Scaling in Large Language Models The Scaling Law [ 16 ] describes the empirical relationship between model performance and scale, including model parameters, data volume, and computational resources. Recent advances in Large Language Models (LLMs), such as GPT-4 [ 1 ] , DeepSeek-R1 [ 12 ] , and Qwen-3 [ 30 ] , have largely benefited from increasing model size. However, training-time scaling faces growing challenges due to rising computational costs and the limited availability of high-quality data. To address these limitations, recent studies have shifted toward Test-Time Scaling