RONA : Pragmatically Diverse Image Captioning with Coherence Relations

Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee · 2025

Read RONA : Pragmatically Diverse Image Captioning with Coherence Relations by Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-10997.txt.

Introduction

Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, and Dongwon Lee wrote a 2025 paper on making image captions more varied in what they mean, not just how they are worded. Writing assistants already diversify captions with synonyms and reshuffled sentences. Human captioners do something else: they choose a message and a stance toward the scene, signaled by pragmatic cues.

Their method, RONA, steers a multimodal language model with coherence relations, the logical links between clauses like cause, contrast, and elaboration. Choosing a different relation reshapes the caption's message while staying true to the image. Across several domains, RONA's captions were more diverse overall and matched reference captions better than baseline prompting. The authors release code for the strategy.

Catalog record

CREATORAashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-10997

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RONA : Pragmatically Diverse Image Captioning with Coherence Relations Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee Abstract Writing Assistants (e.g., Grammarly, Microsoft Copilot) traditionally generate diverse image captions by employing syntactic and semantic variations to describe image components. However, human-written captions prioritize conveying a central message alongside visual descriptions using pragmatic cues. To enhance caption diversity, it is essential to explore alternative ways of communicating these messages in conjunction with visual content. We propose RONA , a novel prompting strategy for Multi-modal Large Language Models (MLLM) that leverages Coherence Relations as a controllable axis for pragmatic variations. We demonstrate that RONA generates captions with better overall diversity and ground-truth alignment , compared to MLLM baselines across multiple domains. Our code is available at: https://github.com/aashish2000/RONA Figure 1: An overview of RONA . The CRs used are Insertion (Ins), Concretization (Con), Projection (Proj), Restatement (Rstt) and Extension (Ext). These relations provide a finite but flexible axis of variation for diverse caption generation compared to instruction-based prompts. 1 Introduction A Writing Assistant (WA) is a tool (e.g., Grammarly, Microsoft Copilot, Copy.ai), often powered by Generative AI, that helps users in various writing tasks. WAs have evolved over the years to support users across a multitude of tasks, with AI-powered assistants being adept at generating a wide selection of content. Image Captioning ( i.e., generating textual descriptions for given images) is one key task that has seen significant advancements with the introduction of Multi-modal Large Language Models (MLLMs). These pre-trained models have achieved remarkable success in generating captions that accurately describe the visual content of images Chen et al. (2024) ; Yue et al. (2023) . However, real-world image captions across different domains often require more than just a description of the visual elements– i.e., they need to convey a central message, provide context, and offer different perspectives on the image Federico (2016) . This leads to significant lack of diversity in the generated captions , limiting the utility of WAs. Existing approaches to fostering diversity in image captions have primarily focused on providing a richer vocabulary ( i.e., syntactic variations) or selecting different components of the image to emphasize ( i.e., semantic variations) Bugliarello and Elliott (2021) ; Li et al. (2022) . Although these methods have shown promise, they often fail to capture the nuanced ways in which humans communicate through captions as shown in Figure 1 . Pragmatic variations, such as multi-modal implicatures and metaphors Genovesi (2020) , which utilize meanings or connotations beyond the literal description of the image, are often employed by caption writers to make their messages more engaging and relatable Weiland et al. (2015) . To address this challenge, in this work, we propose RONA ( R elation-based c O here N ce-aware c A ptioning), a novel prompting strategy for MLLMs inspired by the concept of Coherence Relations (CRs). Based on the principles of Discourse Theory, CRs provide a structured overview of image-text relationships Hobbs (1978) ; Kress (2009) ; O’Halloran et al. (2014) , modeling both contextual and pragmatics aspects of language Ma et al. (2025) ; Mavridou et al. (2015) . We evaluate their effectiveness in image captioning by using CRs as guidelines for generating captions that fulfill specific communicative functions while preserving semantic coherence. Our analysis includes popular MLLMs: Claude-3.5 Sonnet V2 Anthropic () and GPT-4o OpenAI et al. (2024) on two datasets– i.e., news captions (ANNA) and social media captions (Tweet Subtitles). These datasets contain a wide range of visual objects and abstractive captions Anantha Ramakrishnan et al. (2024) , making this a challenging task for MLLMs. We demonstrate that RONA outperforms existing baselines on caption diversity while retaining ground truth similarity. Our contributions are as follows: • We propose RONA , a novel prompting strategy that leverages Coherence Relations (CRs) to generate pragmatically diverse image captions. • We demonstrate that RONA outperforms existing MLLM baselines in terms of diversity and ground-truth similarity on news and social media captioning datasets. • Our analysis shows the viability of CRs to be utilized as an axis of variation for Captioning-based writing assistants. 2 Related Work Writing Assistants MLLMs have enabled WAs to support a variety of writing tasks, with their input ranging from sentence-level suggestions Gero et al. (2022) to long-form writing tasks such as literature reviews and creative writing Choe et al. (2024) , Singh et al. (2023) . In order to improve the Human-AI collaboration experience, there exists a need to incorporate human values into AI-based writing systems and vice versa Shen et al. (2024) , Lee et al. (2024) . Although these studies have focused on broader task domains, specific writing tasks such as caption writing have been less explored Ng et al. (2024) , particularly pragmatically diverse captioning which we aim to address. Image Captioning Traditional Captioning models build on task-specific generative architectures to generate faithful and diverse descriptions for images Mahajan and Roth (2020) ; Liu et al. (2019) . However, with the introduction of MLLMs, general-purpose models capable of multi-modal representations are utilized for caption generation Radford et al. (2021) ; Li et al. (2023) . To improve the alignment between image-text linkages, Coherence Relations (CR) Alikhani et al. (2020) have been leveraged across different downstream text generation tasks Alikhani et al. (2019) ; Vempala and Preoţiuc-Pietro (2019) ; Sosea et al. (2021) . Although popular MLLMs such as GPT-4o and Claude Sonnet 3.5 V2 are poor at predicting and verifying these relationships Thrush et al. (2022) ; Anantha Ramakrishnan et al. (2025) , existing research does not explore the production capabilities of these models. In our work, we investigate the ability of MLLM-based WAs to utilize CRs as a guidance mechanism for in-context learning. 3 Methodology Coherence Relations RONA leverages in-context explanations of Coherence Relations (CRs) as guidance for generating pragmatically diverse captions. We utilize CRs that characterize both entity-level and scene-level linkages between an image and its expected caption Xu et al. (2022) . Entity-level relations describe the relationships between specific objects in the image and their corresponding elaboration in the caption. Scene-level relations, on the other hand, capture the overall context and narrative of the image, providing a broader understanding of the visual content. The selection of these relations are motivated by their generalizability across different domains. Overall, the 5 types of relations that we utilize are: • Insertion : An entity-level relation that describes a type of pragmatic ellipsis , i.e., where the focal object described in the image and caption does not have an explicit mention in the caption. • Concretization : An entity-level relation that utilizes an anchor object which is prominently referenced in the image and caption, with the caption providing additional meaning about its context. • Projection : An entity-level relation where the caption’s description revolves around a particular topic, but this topic is not directly featured in the image. Alternatively, the image contains objects that can be associated to this topic instead, forming an implied link between modalities. • Restatement : A scene-level relation that describes the overall context of the image, with the caption providing a more detailed description of the visual scene. • Extension : A scene-level relation in which the caption elaborates further on the

Read the complete text