Panoptic Captioning: Seeking An Equivalency Bridge for Image and Text
Kun-Yu Lin, Hongjun Wang, Weining Ren, Kai Han · 2025
Read Panoptic Captioning: Seeking An Equivalency Bridge for Image and Text by Kun-Yu Lin, Hongjun Wang, Weining Ren, Kai Han (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-16334.txt.
Introduction
Kun-Yu Lin and co-authors define panoptic captioning, the task of writing the smallest text that fully stands in for an image, in a 2025 paper. A panoptic caption must name every entity, its place and traits, the relations among entities, and the scene's overall state. Leading multimodal models, tested broadly, handle this poorly.
The team answer with two tools. PancapEngine builds training data by detecting entities with a tuned suite, then prompting captions that know those entities. PancapChain splits the hard task into stages and writes the caption step by step. Their 13-billion-parameter model beat much larger open models and even topped GPT-4o and Gemini-2.0-Pro on their new test set and score, PancapScore. The authors offer the task as a bridge task between vision and language.
Catalog record
| CREATOR | Kun-Yu Lin, Hongjun Wang, Weining Ren, Kai Han |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 68 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-16334
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Panoptic Captioning: Seeking An Equivalency Bridge for Image and Text Kun-Yu Lin, Hongjun Wang, Weining Ren, Kai Han Abstract This work introduces panoptic captioning , a novel task striving to seek the minimum text equivalence of images. We take the first step towards panoptic captioning by formulating it as a task of generating a comprehensive textual description for an image, which encapsulates all entities, their respective locations and attributes, relationships among entities, as well as global image state. Through an extensive evaluation, our work reveals that state-of-the-art Multi-modal Large Language Models (MLLMs) have limited performance in solving panoptic captioning. To address this, we propose an effective data engine named PancapEngine to produce high-quality data and a novel method named PancapChain to improve panoptic captioning. Specifically, our PancapEngine first detects diverse categories of entities in images by an elaborate detection suite, and then generates required panoptic captions using entity-aware prompts. Additionally, our PancapChain explicitly decouples the challenging panoptic captioning task into multiple stages and generates panoptic captions step by step. More importantly, we contribute a comprehensive metric named PancapScore and a human-curated test set for reliable model evaluation. Experiments show that our PancapChain-13B model can beat state-of-the-art open-source MLLMs like InternVL-2.5-78B and even surpass proprietary models like GPT-4o and Gemini-2.0-Pro, demonstrating the effectiveness of our data engine and method. Project page: https://visual-ai.github.io/pancap/ 1 Introduction Representing images by textual descriptions is a fundamental topic in computer vision and natural language processing fields [ 1 , 2 ] , which benifits various applications, e.g. , cross-modal retrieval [ 3 , 4 ] , multi-modal learning [ 5 , 6 , 7 , 8 ] , safe content generation [ 9 , 10 ] . While prior works have explored various image caption formats, identifying the most effective format remains an open challenge. The most concise captions, which describe only primary entity categories, often sacrifice critical details like entity attributes. Conversely, highly detailed representations, such as paragraphs detailing all pixel-level semantics and their interrelations, are computationally burdensome due to their length. Inspired by these considerations, this work conceives of finding the minimum text equivalence of an image, an ambitious yet challenging goal, which aims to develop a concise textual description that comprehensively captures its essential semantic elements. Conceptually, achieving minimal text equivalence for images can be seen as aligning images and text in the data space, while existing image-text alignment models like CLIP [ 11 ] perform this in the embedding space. Such text representations would maximize the utility of image information for learning and downstream applications. Figure 1: Top Left: An example to demonstrate our proposed panoptic captioning task, which is formulated as generating a comprehensive textual description encapsulating all entities , their respective locations and attributes , relationships among entities, as well as global image state for a given image. Top Right: We report several models’ performance on four distinct dimensions on the validation and test sets of our SA-Pancap benchmark. The figure shows that three state-of-the-art Multi-modal Large Language Models (MLLMs) struggle with panoptic captioning, while our proposed PancapChain performs generally better with a significantly smaller model size. Bottom: Image “reconstruction” by the text-to-image model PixArt- \Sigma [ 12 ] with different types of captions. This work introduces the task of panoptic captioning , which strives to seek the minimum text equivalence of images. Our work serves as the initial effort towards this challenging task. To make the problem tractable, we formulate panoptic captioning as the task of generating a comprehensive textual description for an image, which captures all entity instances, their respective locations and attributes, relationships among entity instances, as well as global image state (see Figure 1 (top left) ). This formulation serves as a reasonable approximation to our conceptual “minimum text equivalence”, which captures basic semantic elements ( e.g. , center dog) for comprehensiveness while excluding less critical or subtle details ( e.g. , tiny particles on the ground) for conciseness. In contrast to prevailing captioning works that vaguely specify locations by pure words [ 13 , 1 , 2 , 4 , 14 , 15 , 5 ] , e.g. , BLIP-2’s brief captions and ShareGPT4V’s detailed captions, our panoptic captioning stands out for its text comprehensiveness. By accurately localizing entity instances using bounding boxes, it offers an effective and efficient way to describe position and occupied region of an entity instance with only several coordinate numbers. Figure 1 (bottom) shows our panoptic captioner performs better image “reconstruction” from captions due to better comprehensiveness. Due to the fundamental formulation differences from existing captioning works, their metrics cannot effectively evaluate model performance in panoptic captioning. To this end, we propose a new metric named PancapScore, which groups semantic content into distinct dimensions based on task formulation and then conducts evaluation on each dimension. PancapScore first evaluates semantic tagging and instance localization by entity instance matching, and then assesses attribute, relation and global state in a flexible question-answering manner. Our experiments demonstrate that PancapScore aligns closely with human judgement. Based on PancapScore, we conduct a thorough evaluation on state-of-the-art Multi-modal Large Language Models (MLLMs). As shown in Figure 1 (top right) , existing MLLMs have limited performance in panoptic captioning, revealing its challenging nature. To address this new and challenging task, we propose an effective data engine named PancapEngine to produce high-quality data in a detect-then-caption manner. Specifically, we first detect diverse categories of entities in images, by associating class-agnostic detection with image tagging. We then employ state-of-the-art MLLMs to generate comprehensive panoptic captions using entity-aware prompts, while ensuring data quality by caption consistency across MLLMs. Based on PancapEngine, we contribute a new SA-Pancap benchmark composed of high-quality auto-generated data for training and validation, and additionally provide a human-curated test set for reliable evaluation. Furthermore, we propose a novel method named PancapChain to improve panoptic captioning. The key idea is to decouple the challenging panoptic captioning task into multiple stages and train the model to generate panoptic captions step by step. PancapChain first localizes entity instances, then assigns semantic tags to instances, and finally generates panoptic captions. Surprisingly, our PancapChain-13B model beats state-of-the-art large open-source MLLMs like InternVL-2.5-78B, and even surpasses proprietary models like GPT-4o and Gemini-2.0-Pro. Additionally, our model can facilitate downstream image-text retrieval tasks by transforming images into captions and performing text retrieval. Table 1 shows that our model achieves superior performance on the challenging DOCCI dataset, which outperforms a state-of-the-art image-text alignment model ALIGN without specialized training data and module designs, and beats the state-of-the-art detailed captioner ShareGPT4V. These Table 1: Image-text retrieval results on DOCCI [ 4 ] in Recall@1. Models Model Type R@1 ALIGN [ 16 ] Image-Text 59.9 BLIP [ 13 ] Text-Text 47.3 ShareGPT4V [ 5 ] Text-Text 59.6 PancapChain Text-Text 61.9 experiments in panoptic captioning and downstream image-text retrieval demonstrates the effectiveness and application value of our task and model. Our contributions are summarized as follows: First , we introduce the novel panoptic captioning task, which strives to seek the minimum text equivalence of an image – an ambitious yet challenging goal. We formulate it as the task of generating a comprehensive textual description composed of five distinct dimensions, and contribute a comprehensive PancapScore metric for reliable evaluation. Second , we