URECA : Unique Region Caption Anything
Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung, Seungryong Kim · 2025
Read URECA : Unique Region Caption Anything by Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung, Seungryong Kim (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-05305.txt.
Introduction
Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung and Seungryong Kim build URECA, a system and dataset for naming what makes one image region distinct. Region captioning should describe a chosen part, not just the main object. Past sets and models often fail to keep captions unique across fine levels of detail.
Their dataset covers objects, parts and background bits, each tied to one caption through a stage-wise curation pipeline that uses multimodal large language models. The URECA model adds dynamic mask modeling and a high-resolution mask encoder to keep position and shape. It reached top results on their dataset and held up well on older region captioning tests.
Catalog record
| CREATOR | Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung, Seungryong Kim |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-05305
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
URECA : Unique Region Caption Anything Sangbeom Lim, Junwan Kim, Heeji Yoon, Jaewoo Jung, Seungryong Kim Abstract Region-level captioning aims to generate natural language descriptions for specific image regions while highlighting their distinguishing features. However, existing methods struggle to produce unique captions across multi-granularity, limiting their real-world applicability. To address the need for detailed region-level understanding, we introduce URECA dataset, a large-scale dataset tailored for multi-granularity region captioning. Unlike prior datasets that focus primarily on salient objects, URECA dataset ensures a unique and consistent mapping between regions and captions by incorporating a diverse set of objects, parts, and background elements. Central to this is a stage-wise data curation pipeline, where each stage incrementally refines region selection and caption generation. By leveraging Multimodal Large Language Models (MLLMs) at each stage, our pipeline produces distinctive and contextually grounded captions with improved accuracy and semantic diversity. Building upon this dataset, we present URECA, a novel captioning model designed to effectively encode multi-granularity regions. URECA maintains essential spatial properties such as position and shape through simple yet impactful modifications to existing MLLMs, enabling fine-grained and semantically rich region descriptions. Our approach introduces dynamic mask modeling and a high-resolution mask encoder to enhance caption uniqueness. Experiments show that URECA achieves state-of-the-art performance on URECA dataset and generalizes well to existing region-level captioning benchmarks. Figure 1 : Unique Region Caption Anything. We introduce URECA dataset, a novel region-level captioning dataset designed to ensure caption uniqueness and support multi-granularity regions. Each caption in our benchmark is uniquely mapped to its corresponding region, capturing distinctive attributes that differentiate it from surrounding areas. Moreover, we show that our proposed model trained on our dataset effectively generates unique captions for regions at any level of granularity. * * footnotetext: These authors contributed equally. $\dagger$ $\dagger$ footnotetext: Corresponding author. 1 Introduction Region-level captioning aims to describe a specific region in natural language while highlighting its distinguishing features compared to other regions. Although previous approaches have shown strong performance in describing target regions, they have struggled to generate distinguishable captions across multiple granularities . For real-world applications, it is crucial to describe regions at any level of granularity. Despite its importance, generating unique captions across multiple levels of granularity remains underexplored. The primary challenges in multi-granularity captioning include accurately localizing the user-specified region and capturing its unique attributes (e.g., color, relative position, and shape) to clearly distinguish it from surrounding areas [ 50 , 30 , 47 ] . Furthermore, generating truly unique regional captions [ 59 , 35 ] is particularly challenging, as most existing methods [ 20 , 36 , 39 , 57 , 61 , 66 , 19 , 65 ] primarily describe target regions without sufficiently capturing their distinctive attributes regarding their contextual surroundings. Additionally, even though several studies [ 7 , 29 , 28 , 42 ] have explored regional captioning, we find that they struggle to produce truly unique and context-aware captions due to limitations in both model design and dataset availability. Regarding model design, previous approaches [ 51 , 9 , 31 , 63 ] have explored generating captions on salient regions by providing region coordinates through natural language descriptions [ 29 , 5 ] , marking regions directly on the image (e.g., mask contours, ellipses, bounding boxes, triangles, scribbles, points, arrows, and masks) [ 4 , 55 , 41 ] , or utilizing visual RoI features [ 12 , 16 ] . Although these methods have demonstrated notable performance, they come with several drawbacks on solving unique multi-granularity region captioning: they may alter the original image, make it difficult to distinguish between colored contours, and fail to capture global relationships within the image [ 43 ] , ultimately hindering the model’s ability to generate unique captions. On the other hand, we show that the naïve use of existing captioning datasets [ 31 , 61 , 39 , 22 , 56 , 38 , 14 , 18 , 49 , 45 , 6 , 15 ] is also incompatible with unique multi-granularity captioning. Existing datasets primarily focus on salient regions [ 56 , 39 , 49 , 45 ] or rely on bounding box annotations [ 31 , 14 ] , which limits the model’s ability to describe less salient properties. Additionally, most captions in these datasets tend to be overly general [ 22 , 61 , 38 ] (e.g., “a person standing”), leading to duplicate annotations for multiple regions within the same image. This lack of specificity prevents models from distinguishing similar regions effectively, ultimately hindering their ability to generate faithful descriptions of user-defined regions. As current datasets have not explored on both generating unique caption and considering multi-granularity, our paper first presents a novel data curation pipeline on generating unique captions on multi-granularity regions. To generate unique captions for regions while considering their hierarchical relationships, our automated data curation pipeline annotates regions in a stage-wise manner using a mask tree structure. The mask tree captures hierarchical dependencies by representing subset and superset relationships among regions, as shown in Figure 2 . By leveraging this structure, our pipeline efficiently generates unique captions for target regions based on their corresponding tree nodes. Utilizing our data pipeline, we propose URECA dataset , a dataset tailored for unique captioning on multi-granularity regions. Unlike previous captioning datasets [ 31 , 61 , 39 , 22 , 56 , 38 , 14 , 18 , 49 , 45 , 6 ] which are solely limited to salient regions within an image and contain simple descriptions, our dataset covers a broader range of objects, parts, and backgrounds each with a unique caption. Each caption in our dataset is uniquely mapped to a single region, ensuring a unique correspondence between regions and captions across various granularities. Based on our dataset, we then propose a novel model architecture URECA , which effectively conditions regions of interest for the captioning model without losing the region’s details. To consider multi-granularity regions, it is essential to encode various size of regions, which previous methods often struggled with [ 18 , 12 , 60 , 65 , 4 ] . To achieve this, we introduce a mask encoder network and dynamic mask modeling approach that extracts mask features while preserving essential properties such as position and shape even on the multi-granularity regions. Dataset Simple caption Dense caption Region caption Multi-granularity Unique caption RefCOCOg [ 56 ] ✓ ✗ ✓ ✗ ✗ Visual Genome [ 22 ] ✓ ✗ ✓ ✗ ✗ PACO [ 38 ] ✓ ✗ ✓ ✗ ✗ Partimagenet [ 6 ] ✓ ✗ ✓ ✗ ✗ PRIMA [ 45 ] ✓ ✓ ✗ ✗ ✗ LLaVA-115K [ 29 ] ✓ ✓ ✗ ✗ ✗ Arcana [ 42 ] ✓ ✓ ✓ ✗ ✗ Osprey [ 61 ] ✓ ✓ ✓ ✗ ✗ I Dream My Painting [ 10 ] ✓ ✓ ✓ ✗ ✗ GRIT [ 36 ] ✓ ✓ ✓ ✗ ✗ LiSA [ 23 ] ✓ ✓ ✓ ✗ ✗ USE [ 49 ] ✓ ✓ ✗ ✓ ✗ SegCAP [ 67 ] ✓ ✓ ✓ ✓ ✗ GranD [ 39 ] ✓ ✓ ✓ ✓ ✗ URECA dataset (Ours) ✓ ✓ ✓ ✓ ✓ Table 1 : Statistical comparison of