From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models Journal: Information Fusion

Haoxiang Sun, Tao Wang †, Li Yuan, Jian Zhao, Jiancheng Lv · 2026

Read From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models Journal: Information Fusion by Haoxiang Sun, Tao Wang †, Li Yuan, Jian Zhao, Jiancheng Lv (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-26196.txt.

Introduction

Haoxiang Sun, Tao Wang, Li Yuan, Jian Zhao, and Jiancheng Lv survey how multimodal large language models perceive the world. Such models fuse vision and language, and recent releases pushed them toward perception-centred intelligence. Older reviews treat sight and language apart. A unified view, where the two form one inseparable sense, has been missing. Their survey claims that gap as its territory.

Perception gets a formal definition first: one built-in vision-language capacity, much like innate human perception. A five-stage taxonomy then traces how that capacity evolved across model generations. Representative methods and milestones fill each stage. Open challenges close the survey, with research directions toward general unified multimodal intelligence. The authors aim to give the field both a foundation and a roadmap. The venue is a journal devoted to information fusion, fitting for a study of merged senses.

Catalog record

CREATORHaoxiang Sun, Tao Wang †, Li Yuan, Jian Zhao, Jiancheng Lv
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE82 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.26196

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models Journal: Information Fusion Haoxiang Sun, Tao Wang †, Li Yuan, Jian Zhao, Jiancheng Lv Abstract Multimodal Large Language Models (MLLMs) have recently made remarkable progress in unifying vision-language understanding and reasoning, especially following the introduction of models such as OpenAI’s O-series and DeepSeek’s R-series, which have driven a paradigm shift toward perception-centric intelligence. However, there remains a lack of systematic surveys that examine perception from a truly unified vision-language perspective—one that treats vision and language as an inseparable modality. Existing reviews are often fragmented, focusing separately on either vision or language, and thus rarely capture the cross-modal evolution of perception as an integrated capability. To bridge this gap, we present the first systematic survey of unified vision-language perception in MLLMs. Specifically, we (1) formalize MLLM perception as an intrinsic, unified vision-language capability analogous to human innate perception, (2) introduce a five-stage taxonomy tracing the paradigm evolution of MLLM perception and survey representative methods and milestones at each phase, and (3) identify open challenges and outline promising research directions toward truly general, unified multimodal intelligence. We hope our study will provide both a foundational understanding and an actionable roadmap to foster further innovation on the path toward artificial general intelligence (AGI). 1 Introduction “Semantic knowledge transforms the sensory cacophony into a symphony of meaning.” — Matthew A. Lambon Ralph In recent years, Multimodal Large Language Models (MLLMs) 1 1 1 In this paper, “MLLM” is used as a unified term encompassing Large Language Models (LLMs), Large Vision-Language Models (LVLMs) and multimodal LLMs. Our focus is on their unified vision-language perception capabilities. have evolved from a pure text-processing paradigm to unified vision-language understanding and reasoning, significantly advancing cross-modal tasks such as image captioning [ 156 , 13 ] visual grounding [ 194 , 71 ] , and multimodal question answering [ 96 , 109 ] . The perception capability of MLLMs—the ability to extract, localize, and reason about structured semantic information from raw visual signals—serves as the cornerstone for vision-language understanding and reasoning. It underpins applications ranging from classic computer vision benchmarks [ 194 , 88 ] to open-ended, human-centric tasks [ 175 , 17 ] . This perception capability enables MLLMs to see, comprehend, and interact with multimodal information in a human-like manner [ 124 ] . Despite remarkable advances and rapid evolution in MLLMs, there remains a lack of systematic surveys that approach these developments from the perspective of perception. Existing reviews are typically siloed, focusing on either vision-centric [ 130 , 138 ] or language-centric [ 166 , 21 ] tasks, and thus fail to address perception as a unified vision-language capability. This fragmented perspective overlooks the paradigm-level transitions in vision-language perception within MLLMs, hindering a holistic understanding of their full potential and underscoring the need for an integrated, perception-focused review. To address this gap, our survey provides the first in-depth and structured analysis of the evolution of vision-language perception in MLLMs. We clarify the scope of our review, outline its organization, and summarize our main contributions, aiming to serve as a roadmap for future research in unified vision-language perception. 1.1 Scope and Definitions This section defines the scope of our survey, focusing on how MLLMs perceive images at the level of specific regions or instances in response to natural-language queries . Concretely, a method falls within our scope if it satisfies the following: 1. The input includes at least one natural 2D image or video clip. 2. The core challenge lies in extracting or distinguishing visual evidence (objects, regions, attributes, relations), rather than in purely symbolic or mathematical reasoning. 3. Solving the task requires correctly perceiving specific regions or instances under natural-language guidance (e.g., referring expression comprehension, visual grounding, and region-level question answering). Naturally, we explicitly exclude three categories of work: (1) tasks primarily driven by logical or symbolic reasoning rather than perceptual processing (e.g., mathematical problem solving [ 27 ] ); (2) methods targeting highly specialized applications in vertical domains (e.g., robotic vision–language control [ 150 ] ); and (3) approaches whose gains arise solely from global architectural scaling (e.g., increasing input resolution [ 42 ] or model parameter count [ 91 ] ) without introducing mechanisms that enhance localized perception. 1.2 Paper Organization We categorize the evolution of perception capability into five stages: from structure-driven modular optimization (Stages I and II), to dynamic perception paradigms (Stage III), and ultimately to unified frameworks that synergize instruction, adaptivity, and reinforcement learning (Stage IVA and Stage IVB), offering prospects for the next generation of multimodal intelligence. For an intuitive overview, the organizational framework of our survey is illustrated in Fig. 2 . Specifically, the rest of this survey is organized as follows. Sec. 2 (Preliminary) introduces our five-stage taxonomy and contrasts our unified perception focus with prior, task-specific surveys. Sec. 3 (Stage I: Encoder-Centric Optimization) reviews early efforts to strengthen local perception via Region-Aware Modules and Integrated Perception Subnetworks. Sec. 4 (Stage II: Decoder-Centric Optimization) examines how auxiliary decoding shifts perception from region-level to pixel-level through Auxiliary Decoders, Multi-Decoder Architectures, and Specialized Decoding Strategies. Sec. 5 (Stage III: Dynamic Perception via Adaptive Processing) explores adaptive, context-driven perception across External Tool Scheduling, LLM-Centric Routing, and Code-as-Policy for Perception Tasks. Sec. 6 (Stage IVA: Architecture-Free Strategies for Perception Enhancement) delves into two non-architectural approaches: Instruction-Based Strategies and RL-Based Strategies. Sec. 7 (Stage IVB: Towards Unified Perception) provides a forward-looking perspective on the convergence of instruction, adaptivity, and RL via Why Unification Matters, Signs of Convergence and Towards a New Generation of Perception-Centric Agents. Finally, Sec. 8 offers a concise summary of our key findings and outlines directions for future research. 1.3 Main Contributions 1. First systematic survey: To the best of our knowledge, this is the first survey to systematically analyze vision-language perception in MLLMs as an intrinsic, unified capability, providing an actionable roadmap for future research. 2. Paradigm taxonomy and comprehensive review: We propose a five-stage taxonomy of paradigm evolution in MLLM perception and comprehensively review representative works at each stage, highlighting their key innovations. 3. Emerging challenges and future directions: We provide an in-depth discussion of current challenges and outline promising directions for future development in perception-centric multimodal intelligence. 2 Preliminary 2.1 Taxonomy In this section, we introduce our five-stage taxonomy that characterizes the evolutionary trajectory of perception capabilities in MLLMs. This framework is designed to capture the paradigm shift from structure—with targeted, architecture-driven enhancements—to synergy, where perception emerges from integrated, adaptive, and unified strategies. Our taxonomy begins with encoder-centric, structure-driven approaches developed for specific scenarios such as simple image understanding tasks in Visual Question Answering (VQA), where perception improvements are achieved either by enhancing the internal structure of the vision encoder or by modifying the model architecture to treat the encoder as an integrated perceptual subnetwork. It then moves to decoder-centric strategies, which further advance fine-grained perception by employing specialized decoding designs that increase the granularity of the model’s perceptual capabilities—from region-level to pixel-level understanding. The third stage, dynamic perception, focuses on context-aware and adaptive processing, enabling models to flexibly interact with images multiple times to acquire richer and more comprehensive information. The most recent advances represent two distinct but related directions:

Read the complete text