Beyond Single Frames: Can LMMs Comprehend Implicit Narratives in Comic Strip?

Xiaochen Wang, Heming Xia, Jialin Song, Longyu Guan, Qingxiu Dong, Rui Li, Yixin Yang, Weiyao Luo, Yiru Wang, Yifan Pu, Xiangdi Meng, Wenjie Li, Zhifang Sui · 2025

Read Beyond Single Frames: Can LMMs Comprehend Implicit Narratives in Comic Strip? by Xiaochen Wang, Heming Xia, Jialin Song, Longyu Guan, Qingxiu Dong, Rui Li, Yixin Yang, Weiyao Luo, Yiru Wang, Yifan Pu, Xiangdi Meng, Wenjie Li, Zhifang Sui (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13925.txt.

Introduction

Xiaochen Wang, Heming Xia, and colleagues test whether multimodal models can follow stories told across comic panels in a 2025 study. A comic's meaning lives between frames: time passes, causes land, and jokes pay off in later panels. Benchmarks for vision-language models mostly grade single images, leaving that sequence sense unmeasured.

Their benchmark asks models to comprehend temporal and contextual narratives in image sequences. Results show where current models lose the thread that a child reader keeps easily. Sequence comprehension is a fairer test of the story understanding that assistants need for manuals, medical image series, and video.

Catalog record

CREATORXiaochen Wang, Heming Xia, Jialin Song, Longyu Guan, Qingxiu Dong, Rui Li, Yixin Yang, Weiyao Luo, Yiru Wang, Yifan Pu, Xiangdi Meng, Wenjie Li, Zhifang Sui
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13925

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Beyond Single Frames: Can LMMs Comprehend Implicit Narratives in Comic Strip? Xiaochen Wang, Heming Xia, Jialin Song, Longyu Guan, Qingxiu Dong, Rui Li, Yixin Yang, Weiyao Luo, Yiru Wang, Yifan Pu, Xiangdi Meng, Wenjie Li, Zhifang Sui Abstract Large Multimodal Models (LMMs) have demonstrated strong performance on vision-language benchmarks, yet current evaluations predominantly focus on single-image reasoning. In contrast, real-world scenarios always involve understanding sequences of images. A typical scenario is comic strips understanding, which requires models to perform nuanced visual reasoning beyond surface-level recognition. To address this gap, we introduce StripCipher , a benchmark designed to evaluate the model ability on understanding implicit narratives in silent comics. StripCipher is a high-quality, human-annotated dataset featuring fine-grained annotations and comprehensive coverage of varying difficulty levels. It comprises three tasks: visual narrative comprehension, contextual frame prediction, and temporal narrative reordering. Notably, evaluation results on StripCipher reveals a significant gap between current LMMs and human performance—e.g., GPT-4o achieves only 23.93% accuracy in the reordering task, 56.07% below human levels. These findings underscore the limitations of current LMMs in implicit visual narrative understanding and highlight opportunities for advancing sequential multimodal reasoning. † † footnotetext: * Equal first contribution. † † footnotetext: \dagger Equal second contribution. † † footnotetext: \ddagger Corresponding author. 1 Introduction In the space between the panels, human imagination takes separate images and transforms them into a single idea. — Scott McCloud (1993) Figure 1: An example of three tasks: prediction, comprehension, and reordering from the StripCipher dataset. All tasks are presented as multiple-choice questions, with distractors excluded due to limited context. Comics are a widely accessible medium, appealing to audiences ranging from children to adults. Sophisticated comics heavily harness context cues, cultural references, and visual metaphors to convey layered, implicit meaning Magnussen (2000) ; Manning (1998) ; Pressman (2014) . Generally, children with basic comprehension skills can grasp the logical relationships and underlying meanings of multi-panel comics. LMMs have demonstrated impressive performance across various visual-language tasks such as image captioning Liu et al. (2023a) ; Ghandi et al. (2024) , text recognition Liu et al. (2023b) , visual question answering Lu et al. (2023) ; Zhu et al. (2024) and video understanding Zhang et al. (2023) ; Maaz et al. (2024) . However, the question remains whether these models can truly grasp the implicit narratives embedded in comics—a challenge that goes beyond conventional OCR and text reasoning. Existing comic benchmarks Huang et al. (2016) ; Hong et al. (2024) ; Surikuchi et al. (2024) ; Vivoli et al. (2024) ; Sachdeva and Zisserman (2024) have predominantly focus on instance level tasks like object detection and dialogue generation in text-heavy comics, allowing LMMs to rely on OCR and superficial text processing rather than deep visual comprehension. Recent benchmarks Yang et al. (2024) ; Liu et al. (2024b) evaluate single-frame comics with an emphasis on surface-level interpretation, neglecting the sequential dependencies of multi-frame narratives. Motivated by these limitation, we focus on silent comic strips—compositions devoid of explicit textual dialogue and consisting of 3–8 frames. The transition from single images to sequential data is not just a logical progression—it mirrors real-world perceptual processes and aligns with emerging trends in multimodal research. Video-based LLMs Zhang et al. (2025) ; Maaz et al. (2024) ; Zhang et al. (2024) also preprocess video input by sampling key frames before performing inference. Multi-frame comics occupy a distinctive niche between static images and dynamic videos, conveying complete narratives through a minimal, densely packed sequence of frames. Much like keyframes in video analysis, these structured sequences create a controlled yet challenging environment for evaluating temporal understanding, causal relationships, and contextual reasoning in LMMs. We introduce StripCipher , a novel benchmark designed to assess the implicit temporal-visual reasoning ability of LMMs. StripCipher consists of three challenging tasks: (1) contextual frame prediction task to eval visual contextual reasoning, (2) visual narrative comprehension task to infer overall implicit narrative, moving beyond local cues (3) temporal frames reordering, as shown in Figure 1 . We utilized GPT-4o to preprocessing and generate answers and distractors, then employed human annotators double-verify and remove low-quality entries, resulting in 2,170 samples. The dataset includes classic comics like "Peanuts" as well as recent published "Buni", ensuring diversity. Our comprehensive evaluation of 16 state-of-the-art LMMs on StripCipher reveals a substantial performance gap compared to human capabilities in sequential image comprehension, especially in the frame reordering task. Most notably, GPT-4o achieves only 23.93\% accuracy in the reordering subtask and trails human performance by 30\% in visual narrative comprehension. Further quantitative analysis identifies several key factors affecting the sequential understanding performance of LMMs, highlighting the fundamental challenges for future LMM development. Benchmark Task Seq. #Fra #Seq #Cat HCD Funniness Classification ✗ 50 0 1 MTSD Sarcasm Classification ✗ 9,638 0 3 HUB Matching, Ranking, Explanation ✗ 651 0 1 MORE Sarcasm Explanation ✗ 3,510 0 1 DEEPEVAL Description, Title, Deep Semantics ✗ 1,001 0 6 AutoEval-Video Open-Ended Question Answering ✓ 1,033 327 12 Mementos Description Generation ✓ 8,124 699 1 StripCipher Prediction, Comprehension, Reorder ✓ 3,660 896 6 Table 1: Features and statistics of StripCipher and related datasets. Seq. indicates whether image sequences are included. #Fra denotes the number of frames (sub-images), #Seq is the number of image sequences, and #Cat is the number of image categories. Our key contributions are as follows: We introduce StripCipher , the first benchmark that evaluates LMMs on implicit visual narrative reasoning with silent comics. We propose three novel tasks (narrative comprehension, frame prediction, and reordering) that challenge existing LMMs beyond single-frame comic understanding. We comprehensive evaluat 16 state-of-the-art LMMs, revealing substantial gaps between AI and human capabilities. Figure 2: Schematic diagram of StripCipher dataset construction process including three stages: Image Collection , Data Annotation and Cross Check . Only comprehension task is displayed, as Prediction follows the same process. 2 Related Work Large Multimodal Models LLMs have demonstrated exceptional performance in various natural language understanding and generation tasks Dubey et al. (2024) ; Liu et al. (2024a) ; Ray (2023) . Building on the scaling laws of LLMs, the next generation of LMMs has emerged, utilizing LLMs as their backbone. Several closed-source LMMs Reid et al. (2024) ; Driess et al. (2023) ; Yang et al. (2023) , including GPT-4o Hurst et al. (2024) , have shown remarkable capabilities in handling complex multimodal inputs Fu et al. (2023) ; Li et al. (2023) . Beyond single-image processing, models such as QwenVL2.5 Team (2025) and MiniCPM-o2.6 Yao et al. (2024) can handle multiple images, while video-based LLMs Zhang et al. (2025) ; Maaz et al. (2024) ; Zhang et al. (2024) preprocess video content by sampling key frames before inference. Additionally, state-of-the-art models like Gemini Reid et al. (2024) and frameworks incorporating interleaved Chain-of-Thought Gao et al. (2024) explicitly account for the sequential order of both text and images in their reasoning processes. Our approach extends from single to sequential images, mirroring real-world perceptual processes and aligning with emerging trends in multimodal research. Visual Implicit Meanings Understanding Beyond studies on surface-level image understanding Antol et al. (2015) ; Wang et al. (2022) ; Dong et al. (2022) ; Xia et al. (2023) , recent works have shown that LMMs struggle implicit meaning understanding Desai et al.

Read the complete text