Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions
Tommaso Galliena, Tommaso Apicella, Stefano Rosa, Pietro Morerio, Alessio Del Bue, Lorenzo Natale · 2025
Read Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions by Tommaso Galliena, Tommaso Apicella, Stefano Rosa, Pietro Morerio, Alessio Del Bue, Lorenzo Natale (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08531.txt.
Introduction
Tommaso Galliena and co-authors present a 2025 method that teaches exploring agents to describe objects with stable captions. A robot that moves sees each object from many angles, amid clutter. Caption models often give shifting or wrong labels across views. The authors fine-tune caption models with a three-phase plan.
First, the agent roams a space and gathers noisy image and caption pairs. Next, a large language model merges those into one agreed caption per object. Last, the agreed captions retrain the caption model, with contrastive learning added. The team tested mixes of caption models and roaming policies. Steadier captions would help robots report what they see.
Catalog record
| CREATOR | Tommaso Galliena, Tommaso Apicella, Stefano Rosa, Pietro Morerio, Alessio Del Bue, Lorenzo Natale |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08531
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Embodied Image Captioning: Self-supervised Learning Agents for Spatially Coherent Image Descriptions Tommaso Galliena, Tommaso Apicella, Stefano Rosa, Pietro Morerio, Alessio Del Bue, Lorenzo Natale Abstract We present a self-supervised method to improve an agent’s abilities in describing arbitrary objects while actively exploring a generic environment. This is a challenging problem, as current models struggle to obtain coherent image captions due to different camera viewpoints and clutter. We propose a three-phase framework to fine-tune existing captioning models that enhances caption accuracy and consistency across views via a consensus mechanism. First, an agent explores the environment, collecting noisy image-caption pairs. Then, a consistent pseudo-caption for each object instance is distilled via consensus using a large language model. Finally, these pseudo-captions are used to fine-tune an off-the-shelf captioning model, with the addition of contrastive learning. We analyse the performance of the combination of captioning models, exploration policies, pseudo-labeling methods, and fine-tuning strategies, on our manually labeled test set. Results show that a policy can be trained to mine samples with higher disagreement compared to classical baselines. Our pseudo-captioning method, in combination with all policies, has a higher semantic similarity compared to other existing methods, and fine-tuning improves caption accuracy and consistency by a significant margin. Code and test set annotations available at https://hsp-iit.github.io/embodied-captioning/ Fig. 1 : An agent equipped with an instance segmentation and captioning model navigates the environment. Off-the-shelf captioners predict partially wrong and inconsistent captions across different views of the same object: CoCa [ 1 ] , BLIP2 [ 2 ] , Florence2 [ 3 ] , and ChatGPT [ 4 ] . I Introduction Visual understanding is a fundamental skill to recognize and describe objects in a scene, enabling robots to navigate and interact with the environment [ 5 ] . Despite advances in image captioning [ 1 , 6 , 7 , 8 , 3 ] , a captioner deployed on an autonomous agent often generates wrong or inconsistent descriptions across different views of the same object, especially in the case of occlusions or challenging viewing directions (see Fig. 1 ). Previous works tackled the problem of inconsistent captions from three research directions: agent navigation [ 9 , 10 ] , noisy captions [ 6 , 11 , 12 ] , and caption comprehensiveness [ 13 , 14 ] . Navigation-based captioning methods combine the agent observations at different times during navigation [ 10 ] . However, when the agent moves in complex environments, the generated descriptions can be highly inconsistent for the same object. Methods that generate the most appropriate caption for an image either use the alignment between vision and language to select the description from a set of captions [ 12 ] , or prompt a Large Language Model (LLM) to summarize the captions [ 13 ] . Mentioned methods fail in case of inaccurate descriptions that might be selected or propagated to the captions summary. Compared to previous works, we tackle both the noise presence and the caption coherence as challenges inherently present when a robot navigates a complex indoor environments. To this end, we propose a three-phase framework that combines embodied perception with a self-supervised strategy to retrain the captioner enhancing the performance and increasing the consistency of generated descriptions. In the first phase, the agent equipped with an object detector and a captioner navigates the environment, collecting observed RGB-D images and corresponding object captions. The agent also creates a point cloud representation of the scene and clusters the object instances in 3D, given the output of the object detector. In the second phase, for each 3D segmented object instance, we prompt an LLM using the corresponding captions, their frequency and in-context learning to describe each object in the environment, hence filtering out potentially wrong captions and generating a comprehensive pseudo-label. In the third phase, we specialize the captioner through contrastive learning, maximizing the similarity of representation for different views of the same object while minimizing the similarity with other objects’ representation. The framework design is modular to allow the integration of past and future approaches for exploration, pseudo-caption generation, and environment specialization, fostering fair comparisons at each phase, and offering a foundation to improve caption consistency in challenging environments. The main contributions of this paper are the following: • A modular framework to mine agent observations during autonomous navigation, generate pseudo-caption labels, and fine-tune captioning models in a self-supervised way. • A methodology to generate pseudo-captions from image samples prompting an LLM with caption frequencies and in-context learning, avoiding the time-consuming task of manual annotation of a training dataset. • A fine-tuning strategy using contrastive learning to improve caption consistency across object views. • A performance analysis of the combination of exploration policies, pseudo-captioning methods and captioning models on our manually annotated test set composed of more than 8800 different images, specifically collected to assess the performance with different object views. II Related Works In this section, we review the methods in image captioning from RGB images focusing on agent navigation, noisy captions, and caption comprehensiveness to deal with inconsistent descriptions of the same object from different views. Image captioning. Captioners adapt existing models by using additional inputs such as questions about the image, or bounding boxes [ 8 , 2 , 15 , 16 , 17 , 18 , 19 , 20 , 21 ] or by improving the representations using contrastive learning [ 1 , 22 , 7 , 8 , 3 , 6 ] . To fully exploit the generalization capabilities of foundation models, the majority of methods adjusts the representation of vision-language models fine-tuning tokens or transformer layers, reducing the gap between the pre-trained visual and text representations [ 2 ] . For example, X-InstructBLIP and BLIP2 bridge the representation between the image encoder and the text decoder by learning a set of queries to combine visual features with textual features using a lightweight transformer [ 8 , 2 ] . Other methods like SCA and CAT adapt the SAM architecture [ 23 ] through visual chain-of-thoughts [ 17 ] or additional cross-attention layers to combine the vision and text representation and predict regional captions [ 15 ] . A common approach among captioning methods is to learn the representation through contrastive learning i.e., maximizing the similarity among input modalities such as vision, language, and point cloud [ 1 , 22 , 7 , 8 , 3 , 6 ] . The multimodal alignment of different object representation can be learned either during a pre-training stage enabling downstream tasks such as captioning or classification [ 7 , 8 , 3 , 6 ] , or during the training to ground the captions to the visual features [ 1 , 22 ] . Despite the use of contrastive learning on input modalities, methods fail to provide consistent captions across different views of the same object, generating wrong descriptions in the case of challenging viewpoints. In this work, we apply contrastive learning to induce the model to learn close representations of the same object in the latent space while maintaining a margin distance from the representations of other objects, using triplet loss [ 24 ] . Fig. 2 : Our three-phase pipeline to improve consistency of object captions. The agent navigates