Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity

Abid Ali, Diego Mollá-Aliod, Usman Naseem · 2026

Read Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity by Abid Ali, Diego Mollá-Aliod, Usman Naseem (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-11693.txt.

Introduction

Ali, Mollá-Aliod and Naseem present MM-Eval, a unified tool for scoring multi-mode digests by quality, match and range. Multi-mode summary pairs concise text with salient visuals, but current scoring checks quality, match and range alone. MM-Eval mixes text quality from OpenFActScore and G-Eval for truth, flow and fit. Image-text match comes via MLLM-as-judge, and range via CLIP entropy. Tuned on mLLM-EVAL news, it reveals a text-first ladder where truth drives seen quality. It beats rule mixing rivals.

Catalog record

CREATORAbid Ali, Diego Mollá-Aliod, Usman Naseem
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40606
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.11693

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity Abid Ali, Diego Mollá-Aliod, Usman Naseem Abstract Multimodal Large Language Models (MLLMs) have facilitated Multimodal Summarization with Multimodal Output (MSMO), wherein systems generate concise textual summaries accompanied by salient visuals from multimodal sources. However, current MSMO evaluation remains fragmented: text quality, image–text alignment, and visual diversity are typically assessed in isolation using unimodal metrics, making it difficult to capture whether the modalities jointly support a faithful and useful summary. To address this gap, we introduce MM-Eval , a unified evaluation framework that integrates assessments of textual quality, cross-modal alignment, and visual diversity. MM-Eval comprises three components: (1) text quality, measured using OpenFActScore for factual consistency and G-Eval for coherence, fluency, and relevance; (2) image–text relevance, evaluated via an MLLM-as-a-judge approach; and (3) image-set diversity, quantified using Truncated CLIP Entropy. We calibrate MM-Eval through a learned aggregation model trained on the mLLM-EVAL news benchmark, aligning component contributions with human preferences. Our analysis reveals a text-dominant hierarchy in this setting, where factual consistency acts as a critical determinant of perceived overall quality, while visual relevance and diversity provide complementary signals. MM-Eval improves over heuristic aggregation baselines and provides an interpretable, reference-weak framework for comparative evaluation of multimodal summaries. 1 Introduction Modern information consumption increasingly combines text with visuals, motivating Multimodal Summarization with Multimodal Output (MSMO): the task of generating textual summaries paired with relevant images that enrich and contextualize the content 19 ; 20 . MSMO aims to produce coherent multimodal outputs that balance informativeness with cognitive load 5 , improving user satisfaction through faster gist comprehension via images while preserving detail in text 19 . Despite its promise, MSMO has historically been constrained by the semantic gap: the challenge of aligning low-level visual signals with high-level textual semantics. Recent advances in Multimodal Large Language Models (MLLMs), such as GPT-4V, LLaVA 8 , and Qwen-VL 15 , have helped bridge this gap by supporting joint encoding and reasoning across modalities 1 . However, as generative capabilities improve, a critical bottleneck has emerged: the lack of robust, scalable, and reliable automatic evaluation methods for multimodal outputs 21 . Despite the inherently multimodal nature of MSMO, current evaluation protocols remain fragmented: a limitation we refer to as the “Silo Effect”. Outputs are decomposed and assessed via unimodal metrics, each blind to cross-modal coherence. On the textual side, ROUGE remains dominant. While historically impactful, ROUGE is limited in its ability to capture semantic equivalence or factual consistency 13 ; 18 . Consequently, summaries that superficially match reference texts but contain hallucinated facts can still score highly 6 . On the visual side, Image Precision (IP) evaluates whether the model selects the exact images chosen by annotators, penalizing semantically valid alternatives, and ignoring alignment with textual content (see Section 2 for a detailed discussion). Most critically, current metrics fail to assess the interaction between modalities: a system may score well on ROUGE and image selection independently while producing disconnected multimodal outputs. Figure 1 illustrates these limitations with toy examples. Figure 1: Toy examples showing limitations of current MSMO evaluation. To address this, we introduce MM-Eval , a modular and unified evaluation framework designed to assess multimodal outputs along three key dimensions: quality, alignment, and diversity. Rather than combining existing tools, MM-Eval redefines evaluation criteria to reflect the inherently cross-modal nature of MSMO. The framework consists of three pillars: • Text Quality ( S_{text} ): We adopt a “decompose-then-verify” approach using OpenFActScore 6 to assess atomic factual consistency, and G-Eval 9 , an LLM-based protocol for coherence, fluency, and relevance. • Cross-Modal Alignment ( S_{relevance} ): Using an MLLM-as-a-Judge framework 21 , we measure how well selected images semantically support the generated text, capturing the integrated quality of the multimodal summary. • Visual Diversity ( S_{diversity} ): To discourage visual redundancy, we use Truncated CLIP Entropy (TCE) 3 , which measures the distinctiveness of the image set in latent space. MM-Eval aggregates these components via a learned weighting model trained on the mLLM-EVAL news benchmark. Our analysis of human judgments from this dataset reveals a text-dominant preference structure in the news domain. In news, factual consistency serves as a key determinant of perceived quality, while visual relevance and diversity provide complementary signals whose contribution is conditional on adequate textual fidelity. We validate these findings through a human evaluation study that confirms annotators value all three dimensions, supporting MM-Eval’s multi-pillar design. Since the component scorers are reference-weak and domain-portable, MM-Eval can be applied to new domains by recalibrating only the lightweight aggregation weights with minimal human supervision. This paper details the design of MM-Eval , its component metrics, and the empirical patterns uncovered through alignment with human ratings. 2 Literature Review 2.1 Traditional Evaluation Metrics Text-based Metrics. Summarization evaluation has long relied on ROUGE 7 , which computes n -gram overlap with reference summaries. While effective for extractive systems, ROUGE cannot capture semantic equivalence, paraphrasing, or factual errors, and penalizes bridging sentences that refer to accompanying visuals 13 . BERTScore 17 addresses lexical rigidity via contextual embeddings but remains reference-based and fails to distinguish factual from hallucinated content 18 ; 14 . Image-based Metrics. Image selection is typically evaluated using Image Precision (IP) and Image Recall (IR), which check whether the model selects the exact annotator-chosen images 20 . This assumes a single correct reference set: if a model selects a semantically equivalent but different image, it receives no credit. Moreover, redundant images in the reference set may inflate IP without adding informational value. Multimodal Aggregation. The Multimodal Automatic Evaluation (MMAE) framework 19 aggregates ROUGE-L, IP, and image–text cosine similarity via a regression model trained on human satisfaction scores. While the aggregation principle is sound, reliance on ROUGE and IP retains the shortcomings mentioned above. MM-Eval builds on this idea but incorporates semantically grounded, reference-weak components. 2.2 LLM-based Evaluation Approaches LLM-as-a-Judge. Recent work such as mLLM-Eval 21 uses large multimodal models (e.g., GPT-4V) to rate summaries directly via Chain-of-Thought prompting. While effective, this approach is resource-intensive and acts as a black box with no explicit quality breakdown. MM-Eval addresses these gaps by decomposing evaluation into interpretable, modular sub-metrics built on open-source models, allowing individual components to be independently replaced or upgraded. Fact-level and Diversity Evaluation. FActScore 10 proposes a “decompose-then-verify” approach, breaking summaries into atomic facts validated against the source. OpenFActScore 6 extends this with open-source models. For visual diversity, Truncated CLIP Entropy (TCE) 3 computes entropy over eigenvalues of CLIP embeddings, providing a reference-free signal that penalizes redundant visual outputs without requiring large sample sets like FID 2 . The extent to which spectral entropy in CLIP space fully captures human notions of visual diversity remains an open question. 3 Methodology Formally, the input to MM-Eval consists of a source document D=\{T_{\text{source}},V_{\text{source}}\} and a system-generated summary S_{\text{cand}}=\{T_{\text{gen}},V_{\text{sel}}\} . The output is a scalar score S_{\text{final}}\in\mathbb{R} , representing the overall quality of the candidate summary. The score is computed by integrating signals from three core components: (1) textual quality, (2) cross-modal alignment, and (3) visual diversity. Figure 2: An overview of MM-Eval Framework. 3.1 Pillar 1: Quantifying Textual Summary Quality ( S_{text} ) The textual dimension of MM-Eval assesses two core aspects: Factual Consistency :

Read the complete text