ComicsPAP: understanding comic strips by picking the correct panel
Emanuele Vivoli, Artemis Llabrés ⋆, Mohamed Ali Souibgui, Marco Bertini, Ernest Valveny Llobet, Dimosthenis Karatzas · 2025
Read ComicsPAP: understanding comic strips by picking the correct panel by Emanuele Vivoli, Artemis Llabrés ⋆, Mohamed Ali Souibgui, Marco Bertini, Ernest Valveny Llobet, Dimosthenis Karatzas (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08561.txt.
Introduction
Comic strips tell stories through sequence, and a missing panel can only be found by following plot, time, and space together. Emanuele Vivoli and co-authors build ComicsPAP in a 2025 paper, a large test of that skill for multimodal AI models. The set holds over one hundred thousand samples in five subtasks.
Each task asks the model to pick the panel that belongs in a gap. Top current models scored near chance, whether the panels came as many images or one strip. The gap shows how weakly models still track sequence and context. After adaptation, the models of the team beat systems ten times their size, which makes the benchmark a useful training ground.
Catalog record
| CREATOR | Emanuele Vivoli, Artemis Llabrés ⋆, Mohamed Ali Souibgui, Marco Bertini, Ernest Valveny Llobet, Dimosthenis Karatzas |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08561
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ComicsPAP: understanding comic strips by picking the correct panel Emanuele Vivoli, Artemis Llabrés ⋆, Mohamed Ali Souibgui, Marco Bertini, Ernest Valveny Llobet, Dimosthenis Karatzas Abstract Large multimodal models (LMMs) have made impressive strides in image captioning, VQA, and video comprehension, yet they still struggle with the intricate temporal and spatial cues found in comics. To address this gap, we introduce ComicsPAP , a large-scale benchmark designed for comic strip understanding. Comprising over 100k samples and organized into 5 subtasks under a Pick-a-Panel framework, ComicsPAP demands models to identify the missing panel in a sequence. Our evaluations, conducted under both multi-image and single-image protocols, reveal that current state-of-the-art LMMs perform near chance on these tasks, underscoring significant limitations in capturing sequential and contextual dependencies. To close the gap, we adapted LMMs for comic strip understanding, obtaining better results on ComicsPAP than 10x bigger models, demonstrating that ComicsPAP offers a robust resource to drive future research in multimodal comic comprehension. Figure 1 : ComicsPAP dataset composition (left) and our Comic-adapted LMM performances vs. zero-shot LMMs (right). 1 Introduction Large multimodal models (LMMs) have achieved remarkable progress in understanding diverse data modalities, excelling at tasks such as image captioning [ 4 , 1 ] , visual question answering (VQA) [ 1 , 2 , 27 , 4 ] , and video comprehension [ 17 ] . These success stories share a crucial advantage: the underlying data are naturally segmented (e.g., video frames) and exhibit strong temporal or spatial coherence, easing the burden of cross-frame reasoning. Comic strips, by contrast, deliberately disrupt these assumptions. Each panel depicts a distinct moment, producing abrupt visual shifts that break the smooth continuity exploited by video models. Furthermore, segmentation is implicit—readers infer panel boundaries from gutters whose presence, shape, and clarity are highly variable. Linguistic signals are likewise fragmented: dialogue and narration appear in localized bubbles or captions and must be stitched together across non-adjacent panels. Effective comic comprehension therefore hinges on three intertwined capabilities: (i) inferring latent panel boundaries, (ii) resolving recurring characters under diverse artistic styles, and (iii) reconstructing causal story lines from sparse, discontinuous multimodal evidence. These challenges position comics as a rigorous stress test for next-generation vision–language models. Recent studies have begun to explore the intricacies of comics understanding. A comprehensive survey [ 23 ] has highlighted both established and emerging tasks aimed at modeling and understanding comics using machine learning techniques. While advanced tasks such as comic story comprehension and multi-page comic generation have been proposed, even more fundamental challenges—such as single-page understanding [ 14 , 13 , 19 ] and single-panel generation [ 25 ] —remain challenging for state-of-the-art vision-language models. Despite progress in object detection [ 21 , 14 ] , full-page analysis [ 19 ] , and single-panel interpretation [ 20 ] , it is still unclear whether foundational models can comprehend the sequential narrative across multiple panels. In this work, we address the lack of comic strip analysis by benchmarking foundational models for comic panel sequence understanding. Our contributions are threefold: (i) we introduce the ComicsPAP dataset, structured around three distinct skills and five subtasks, which reframes sequence understanding as the task of selecting the appropriate panel that best fits within a given ordered set—an approach we term “Pick A Panel”; (ii) we evaluate state-of-the-art LMMs on ComicsPAP and demonstrate that their performance is no better than random choice; (iii) we fine-tune models with 3B and 7B parameters on ComicsPAP , achieving superior performance compared to larger models on this task. The ComicsPAP dataset is publicly available on HuggingFace 1 1 1 https://huggingface.co/datasets/VLR-CVC/ComicsPAP , as are the fine-tuned model weights 2 2 2 https://huggingface.co/VLR-CVC/Qwen2.5-VL-3B-Instruct-lora-ComicsPAP . The training and evaluation code are available in the GitHub repository 3 3 3 https://github.com/llabres/ComicsPAP , and an evaluation server is hosted on the competition page 4 4 4 https://rrc.cvc.uab.es/?ch=31&com=tasks . 2 Related works Data Availability. Availability and copyrighted data remain the foremost challenges in comics analysis. Many previous studies have addressed these issues by linking to original sources to collect images while only sharing the accompanying annotations [ 21 , 19 ] . However, such approaches face significant limitations when scaled up. For instance, source material may become inaccessible over time, resulting in non-reproducible experimental settings (over 50% of LAION images [ 15 ] are no longer available). An alternative strategy involves providing scripts to download images directly from the original sources [ 14 ] and sharing only cropped versions [ 13 ] . Yet these methods also have drawbacks: sources can expire, and users might lack the necessary rights to download copyrighted images. To overcome these challenges, we exclusively utilize non-copyrighted, American-comic-styled materials from the renowned DCM website 5 5 5 https://digitalcomicmuseum.com/index.php . This choice enables us to share both the images and their annotations freely. Panel Sequence Analysis in Comics. Early work in panel sequence analysis, such as COMICS [ 8 ] , introduced “closure” tasks where models select the correct subsequent panel from options based on visual cues, combined visual and textual features, or character-driven attributes. However, these methods relied on old-generation OCR and panel extraction, resulting in poor-quality annotations and distractors sampled from unrelated pages—issues that rendered the task relatively trivial (with recent transformer approaches yielding already significant improvements). Subsequent work like ComicVT5 [ 22 ] improved OCR accuracy, sampled distractor panels from adjacent pages, and extended the task to text generation rather than classification. More recently, MangaUB [ 7 ] extended these evaluations to manga panels with tasks ranging from single-panel analysis (predicting weather, time, and counting characters) to comic strip understanding, yet even basic inferences proved challenging for LMMs. Complementing these efforts, ComiCap [ 20 ] showed that, with proper model selection, vision-language models can generate detailed captions in a zero-shot setting, underscoring the potential for further advances in comic panel comprehension. Despite these advances, panel sequence analysis has largely stagnated with tasks based on earlier, automatically generated annotations for next-panel classification using randomly sampled alternatives. This approach suffers from several intrinsic limitations: (i) comics are seldom segmented into coherent stories before analysis, so randomly sampled panels often come from different narratives, styles, and character sets, making the task inherently easier; (ii) tasks are predominantly framed as next-panel prediction, neglecting the evaluation of causal reasoning through the reconstruction of sequences with missing panels; and (iii) the focus remains largely visual, with little emphasis on selecting the correct panel based solely on textual descriptions. In this work, we build upon these foundations by creating a dataset designed around specific skill sets for comics comprehension. We systematically identify both existing and novel tasks that highlight these underlying capabilities. Moreover, we benchmark state-of-the-art open-weight LMMs on our ComicsPAP dataset and provide a held-out test set and a validation set, thereby providing the comics and LMM communities with a robust platform for model evaluation. 3 Dataset Creation This section describes the construction and annotation procedures of the ComicsPAP dataset. Our approach is driven by explicitly defining the cognitive skills we aim to assess and designing corresponding tasks that rigorously evaluate these skills. Specifically, we focus on three primary multimodal understanding skills— narrative anticipation , multi-frame reasoning , and co-reference resolution . 3.1 Tasks definition Each skill is operationalized through carefully designed