CORDIAL : Can Multimodal Large Language Models Effectively Understand Coherence Relationships?

Aashish Anantha Ramakrishnan , Aadarsh Anantha Ramakrishnan , Dongwon Lee, The Pennsylvania State University, National Institute of Technology, Tiruchirappalli {aza6352, dul13}@psu.edu , [email protected] · 2025

Read CORDIAL : Can Multimodal Large Language Models Effectively Understand Coherence Relationships? by Aashish Anantha Ramakrishnan , Aadarsh Anantha Ramakrishnan , Dongwon Lee, The Pennsylvania State University, National Institute of Technology, Tiruchirappalli {aza6352, dul13}@psu.edu , [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-11300.txt.

Introduction

Aashish Anantha Ramakrishnan and co-authors test whether AI models grasp how images and text hang together in a 2025 paper. Models ace many factual tasks, but coherence, the way parts of a message relate, is a subtler skill. Their benchmark, CORDIAL, spans many relation types across three kinds of discourse and several levels of detail.

More than ten leading multimodal models were tried with varied prompting styles. Even the strongest, including two famous commercial systems, failed to match simple classifier baselines. Scores based on surface similarity miss what these relations demand. The authors argue evaluation should move toward discourse-aware tests. Their benchmark and code are public for others to use.

Catalog record

CREATORAashish Anantha Ramakrishnan , Aadarsh Anantha Ramakrishnan , Dongwon Lee, The Pennsylvania State University, National Institute of Technology, Tiruchirappalli {aza6352, dul13}@psu.edu , [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-11300

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CORDIAL : Can Multimodal Large Language Models Effectively Understand Coherence Relationships? Aashish Anantha Ramakrishnan , Aadarsh Anantha Ramakrishnan , Dongwon Lee, The Pennsylvania State University, National Institute of Technology, Tiruchirappalli {aza6352, dul13}@psu.edu , [email protected] Abstract Multimodal Large Language Models (MLLMs) are renowned for their superior instruction-following and reasoning capabilities across diverse problem domains. However, existing benchmarks primarily focus on assessing factual and logical correctness in downstream tasks, with limited emphasis on evaluating MLLMs’ ability to interpret pragmatic cues and intermodal relationships. To address this gap, we assess the competency of MLLMs in performing Multimodal Discourse Analysis (MDA) using Coherence Relations. Our benchmark, CORDIAL , encompasses a broad spectrum of Coherence Relations across 3 different discourse domains at varying levels of granularity. Through our experiments on 10+ MLLMs employing different prompting strategies, we show that even top models like Gemini 1.5 Pro and GPT-4o fail to match the performance of simple classifier-based baselines. This study emphasizes the need to move beyond similarity-based metrics and adopt a discourse-driven framework for evaluating MLLMs, providing a more nuanced assessment of their capabilities. The benchmark and code are available at: https://aashish2000.github.io/CORDIAL/ . 1 Introduction Figure 1: CORDIAL presents a combination of literal and pragmatic relations for analyzing the intermodal reasoning capabilities of MLLMs. We evaluate MLLMs on the task of Multimodal Discourse Analysis through the prediction and verification of Coherence Relations across three different discourse domains. The recent advancements in Multimodal Large Language Models (MLLMs) enable them to effectively capture diverse representations of problem domains Alayrac et al. (2022) ; Chen et al. (2024c) ; Pichai (2024) ; Liu et al. (2024a) . These MLLMs are capable of adapting to various downstream tasks with limited data through Parameter-Efficient Fine-Tuning (PEFT) Hu et al. (2021) and In-Context Learning (ICL) Brown et al. (2020) approaches. Existing Vision-based MLLM benchmarks assess different aspects of model performance such as Perception, Cognition, and Reasoning Li et al. (2024) through various downstream tasks. Current benchmark design strategies often focus on evaluating the ability of MLLMs to utilize the intersection of input sources to solve a common problem Kruk et al. (2019) . Although this helps assess the model’s ability to interpret its inputs factually and logically, it does not fully capture the model’s understanding of the relationships between these modalities . Similarly, benchmarks that evaluate the alignment between images and text Thrush et al. (2022) , utilize curated or synthetically generated image-text pairs. These methods focus solely on literal relations that measure the level of overlap between the image and text. On the other hand, pragmatic cues provide information on non-literal relations where the true intent/message of an example may not be directly referenced in both modalities as shown in Figure 1 . These cues are leveraged routinely in real-world multimodal discourses, which are characterized by the use of multiple modes of communication to convey different components of a message. Multimodal Discourse Analysis (MDA) studies how the interaction between these different modes can create semiotic meaning Kress (2009) . To operationalize the assessment of these intermodal relationships, we turn to theories of Discourse Coherence Hobbs (1978) , which offer a way to quantify the organization and flow of ideas across information sources. From these theories, we focus on the concept of Coherence Relations Alikhani and Stone (2019) , which provides a finite structure to link different parts of a discourse. Recent studies have extended these traditionally text-only theories to multimodal discourses, showing that Coherence Relations can be effectively applied to image-text pairs Alikhani et al. (2020) . With Coherence Relations being a fundamental aspect of human communication, we evaluate whether MLLMs can effectively predict and verify these relations. In this work, we propose the CORDIAL ( CO herence R elations in D iscourse for I mages A nd L anguage), the first benchmark for evaluating MLLMs on the task of MDA. CORDIAL consists of a diverse set of Coherence Relations across three different discourse domains: Disaster Management, Social Media, and Online Articles. Each domain also offers different levels of complexity in the evaluated Coherence Relations, from binary relations to more challenging settings such as multi-class and multi-label relations assigned by human annotators. We evaluate the performance of 10+ MLLMs on CORDIAL , focusing on three research questions: RQ1: Can MLLMs predict Coherence Relations effectively? RQ2: Can MLLMs verify Coherence Relations accurately? RQ3: Can we teach MLLMs to understand Coherence Relations better? Our analysis reveals that both Coherence Relation prediction (RQ1) and verification (RQ2) are challenging tasks for MLLMs when these relations focus on pragmatic cues. Although larger MLLMs perform better than their smaller, open-source counterparts, traditional classifier baselines consistently outperform them across discourse domains. To summarize, our key takeaways are as follows: • We propose CORDIAL , the first benchmark for evaluating MLLMs for Multi-modal Discourse Analysis (MDA) using Coherence Relations. • Our experiments show that MLLMs struggle to predict and verify Coherence Relations, especially when these relations are more pragmatic. • We demonstrate the need for coherence-aware fine-tuning approaches to improve intermodal reasoning capabilities of MLLMs. Dataset Examples DisREL Part of my pile of branches after #HurricaneIrma - still no power in #Orlando Floridians rescue stranded manatees as Irma sucks water from shores Coherence Relation: Similar Coherence Relation: Complementary Tweet Subtitles Fresh never frozen jumbo wings tossed in a housemade buffalo sauce. Yum! Freshly picked off my allotment today, well chuffed. (strawberry) Cartel leader whose arrest sparked killings is sentenced to prison in Dallas court Amazon Prime delivers anything these days! (delivering a cat) Eiffel Tower shuts down as snow, freezing rain pummel France Coherence Relation: Concretization Coherence Relation: Insertion Coherence Relation: Projection Coherence Relation: Extension Coherence Relation: Restatement CLUE A path winds through an ancient bamboo forest A model walks the runway for the collection during, fashion week A city in winter is such a beautiful city People know that curb appeal is not a thing to take lightly when remodeling a home Seals fighting for a spot to sleep on the rocks Coherence Relations: Visible Coherence Relations: Visible, Meta , Action Coherence Relations: Subjective , Story Coherence Relations: Story Coherence Relations: Action Table 1: Examples from each dataset for all Coherence Relations. The words in red are important cues present in the caption, while the words in orange show pragmatic cues inferred from the image-text pair. The relations highlighted in blue are the selected relations for CLUE Single-Label. Figure 2: An overview of the Image-Text label (i.e., Coherence Relations) distributions across CORDIAL 2 Related Work Multimodal Large Language Models MLLMs are fundamentally generative models that combine Large Language Models (LLM) Brown et al. (2020) with multimodal encoders Dosovitskiy et al. (2021) . In recent years, several new MLLMs have been released, based on various proprietary OpenAI et al. (2024) ; Anthropic () ; Pichai (2024) and open-source LLM backbones Liu et al. (2023) ; Wu et al. (2024) ; Bai et al. (2023) . These models have shown impressive performance on a variety of downstream reasoning tasks, including Visual Question Answering Wu and Xie (2024) , Document Analysis Lv et al. (2023) , Embodied AI agents Shek et al. (2024) , etc. MLLM Reasoning Benchmarks Recent works that have proposed benchmarks evaluating vision language reasoning, focus on assessing

Read the complete text