Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos
Weifeng Lin Xinyu Wei Ruichuan An Tianhe Ren Tingwei Chen, Renrui Zhang Ziyu Guo Wentao Zhang Lei Zhang Hongsheng Li · 2025
Read Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos by Weifeng Lin Xinyu Wei Ruichuan An Tianhe Ren Tingwei Chen, Renrui Zhang Ziyu Guo Wentao Zhang Lei Zhang Hongsheng Li (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05302.txt.
Introduction
The Perceive Anything Model is set out by Weifeng Lin and co-authors in a 2025 arXiv paper. The model reads regions in photos and videos and gives back names, glosses, use notes, and full captions, with outlines at the same time. It grows a known segment tool by joining a language model. A part called the Semantic Perceiver turns sight traits into tokens the language side can read.
A data clean and boost pipe made millions of region notes for photos and videos. The model runs faster and leaner than past tools of its kind. Scores stay strong across region tasks.
Catalog record
| CREATOR | Weifeng Lin Xinyu Wei Ruichuan An Tianhe Ren Tingwei Chen, Renrui Zhang Ziyu Guo Wentao Zhang Lei Zhang Hongsheng Li |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-05302
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos Weifeng Lin Xinyu Wei Ruichuan An Tianhe Ren Tingwei Chen, Renrui Zhang Ziyu Guo Wentao Zhang Lei Zhang Hongsheng Li Abstract We present Perceive Anything Model (PAM), a conceptually straightforward and efficient framework for comprehensive region-level visual understanding in images and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation with the generation of diverse, region-specific semantic outputs, including categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2’s rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support robust multi-granularity understanding, we also develop a dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region understanding tasks. It runs 1.2 − - 2.4 × \times faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that our effective approach will serve as a strong baseline for future research in region-level visual understanding. Code, model and data are available at: https://Perceive-Anything.github.io Figure 1: Perceive Anything Model (PAM): PAM accepts various visual prompts (such as clicks, boxes, and masks) to produce region-specific information for images and videos, including masks, category, label definition, contextual function, and detailed captions. The model also handles demanding region-level streaming video captioning. 1 Introduction The vision community has rapidly witnessed advances in vision foundation models, such as SAM kirillov2023segment and SAM 2 ravi2024sam , which have dramatically improved interactive object segmentation performance in images and videos. These models offer remarkable precision in localizing arbitrary objects based on various visual prompts. However, they typically lack deep semantic understanding of the segmented regions, elucidating what these regions mean or how they function in context remains a challenging problem. Recent studies seek to endow Vision–Language Models (VLMs) with region-level understanding capability through visual prompts. As illustrated in Fig. 2 , current methods can be grouped into three paradigms: (1) textual encoding wang2024elysium ; yu2024merlin ; zhao2023chatspot ; liu2024sphinx , which encode 2-D bounding-box coordinates as natural-language strings inside the prompt, thereby supplying no explicit region prior; (2) visual-prompt encoding (VPE) lin2024draw ; rasheed2024_Glamm_pixelgroundinglarge , which introduce extra module to embed regional image features and positional features; (3) RoI/segmentation-based encoding lian2025describe ; you2023ferret ; zhang2024ferretv2 ; yuan2024osprey ; jiang2024chatrex , which utilize an external mask generator to concatenate image embedding and mask embedding. While these methods show promise, they often present several limitations: (i) they usually generate only limited semantic outputs—often just category labels or short captions huang2024segment ; zhao2024controlcap ; wu2024grit ; wang2023caption ; (ii) their designs are modality-specificl, focusing on one single visual modality (image or video), offering limited generality wang2024elysium ; yu2024merlin ; you2023ferret ; yuan2024osprey ; yuan2024videorefer . (iii) they rely on external segmentation models to supply masks, a serial design that adds computational overhead and makes overall performance sensitive to mask quality yuan2024osprey ; yuan2024videorefer ; lian2025describe . To address these challenges, we introduce the Perceive Anything Model (PAM), an end-to-end region-level vision-language model designed for fast and comprehensive fine-grained visual understanding across both images and videos, encompassing capabilities such as predicting categories, explaining the definition and contextual function of identified regional elements, and generating detailed descriptions of specific regions. Rather than redesigning model architecture from scratch, our approach efficiently extends the SAM 2 framework with Large Language Models (LLMs) to support semantic understanding. Specifically, we introduce a Semantic Perceiver that acts as a essential bridge, effectively leveraging rich intermediate visual features from the SAM 2 backbone to integrate general vision, localization, and semantic priors into visual tokens. These tokens are subsequently processed by the LLM to generate a diverse semantic outputs. Furthermore, PAM features a parallel design for its mask and semantic decoders, enabling simultaneous generation of region masks and semantic content, thereby improving computational efficiency. To ensure PAM’s robustness in understanding region-level multi-dimensional semantic granularity, high-quality training data is an essential component. While multiple existing datasets RefText ; kazemzadeh2014_RefCOCO3 ; krishna2016_VisualGenome_connectinglanguage ; liu2023gres ; jiang2024chatrex ; wang2023allseeing provide region-semantics annotations, we noticed that they are often overly coarse, limiting their utility for fine-grained understanding tasks. Therefore, to construct high-quality training data, we develop an advanced data refinement and augmentation pipeline that leverages leading VLMs (e.g., GPT-4o hurst2024gpt ) and human expert validation to refine and augment existing region-level annotated datasets. For images, we generate annotations at multiple distinct semantic granularities for each specific region: a fine-grained category label, a context-aware definition that clarifies the region’s role or function within the scene, and detailed descriptions. For videos, we refined original coarse-level annotations from referring video detection and segmentation dataset wang2023_LV_VIS_openvocabularyvideoinstancesegmentation ; tang2021_HC_STVG_humancentricspatiotemporalvideogrounding ; gavrilyuk_A2D_2018actor ; xu2018_YoutubeVOS_largescalevideoobject ; ding2023_Mevis_largescalebenchmarkvideo into detailed, temporally-aware region-level captions. Furthermore, we pioneered the development of event-based, region-level streaming video caption data. To the best of our knowledge, this is the first work to construct such a dataset, enabling the model to support streaming video region captioning. Notably, we also generate bilingual (English and Chinese) versions of each data annotation to equip the model with multilingual response capabilities. This process yields a final high-quality dataset comprising 1.5M image-region-semantics triples and 0.6M video-region-semantics triples. Our experimental results demonstrate that PAM delivers robust performance across a diverse range of regional understanding tasks for both images and videos, while operating 1.2 -2.4\times faster and consuming less GPU memory compared to prior models. We believe our model, dataset, and insights will significantly advance research in this domain and broadly benefit the vision-language community. Figure 2: Previous Paradigms vs. Our Paradigm (PAM). (a & b) Textual/VPE methods provide region understanding using positional embeddings but typically lack simultaneous object masks. (c) RoI/Segmentation-based methods use external segmenter for object masks, subsequently fusing image and mask embeddings. (d) In contrast to previous paradigms, our method directly treats the Seg. model as vision encoder. It effectively leverages the rich visual embeddings from the robust segmentation model and features a parallel design for its mask and semantic decoders. 2 Related Work Interactive Image and Video Object Segmentation. Interactive object segmentation has progressed rapidly in recent years. Early methods—such as Graph Cut boykov2006graph and Active Contours chan2001active —relied on manual annotations (e.g., foreground/background clicks). Inspired by the paradigm of pre-training autoregressive Transformer architectures on large-scale data in language modeling, the Segment Anything Model (SAM) kirillov2023segment revolutionized user–model interaction by ingesting multiple visual prompts and segmenting arbitrary objects in a class-agnostic manner. SAM 2 ravi2024sam extended this capability to video, enabling real-time processing of arbitrarily long sequences with strong generalization. Subsequent research ke2023segment ; zhang2023faster ; zhao2023fast ; ren2024grounded ; cheng2023segment ; yang2024samurai , while retaining the core architecture, has further improved the accuracy and efficiency of this model family. Region-level Vision-Language Models (VLMs). Region-level understanding tasks, such as region classification and captioning, are fundamental in computer vision. Regarding image-based VLMs, recent researches peng2023kosmos ; chen2023shikra ; wu2024visionllm ; zhang2025gpt4roiinstructiontuninglarge ; you2023ferret ; yuan2024osprey ; cai2024vip ; lin2024draw ; an2024mc