Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs
Mohamad Ballout, Okajevo Wilfred, Seyedalireza YaghoubiNohayr Muhammad Abdelmoneim , Julius Mayer , Elia Bruni Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany [email protected] · 2025
Read Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs by Mohamad Ballout, Okajevo Wilfred, Seyedalireza YaghoubiNohayr Muhammad Abdelmoneim , Julius Mayer , Elia Bruni Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-24640.txt.
Introduction
Mohamad Ballout and five co-authors present a 2025 paper introducing SPLICE, a benchmark for visual reasoning in vision-language models. The benchmark draws on instructional videos and asks systems to put shuffled event clips back into a sensible order. Human curators filtered over three thousand videos across twelve categories, from sports to housework, yielding more than eleven thousand clips. Ordering them tests time, cause, space, context, and general knowledge together.
Models scored far below human participants on the task. Adding human-written text descriptions raised model scores but not human ones, a sign models lean on language habits over true visual grasp. Models did better where time and cause dominated than where space and context did. They also handled everyday tasks better than specialized ones. The authors present SPLICE as a hard test that exposes weak spots in visual reasoning.
Catalog record
| CREATOR | Mohamad Ballout, Okajevo Wilfred, Seyedalireza YaghoubiNohayr Muhammad Abdelmoneim , Julius Mayer , Elia Bruni Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-24640
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Can you SPLICE it together? A Human Curated Benchmark for Probing Visual Reasoning in VLMs Mohamad Ballout, Okajevo Wilfred, Seyedalireza YaghoubiNohayr Muhammad Abdelmoneim , Julius Mayer , Elia Bruni Institute of Cognitive Science, Osnabrück University, Osnabrück, Germany [email protected] Abstract In this work, we introduce SPLICE, a human-curated benchmark derived from the COIN instructional video dataset, designed to probe event-based reasoning across multiple dimensions: temporal, causal, spatial, contextual, and general knowledge. SPLICE includes 3,381 human-filtered videos spanning 12 categories and 180 sub-categories, such as sports, engineering, and housework. These videos are segmented into a total of 11,423 event clips. We evaluate both human participants and state-of-the-art vision-language models (VLMs) on the task of rearranging these clips into coherent event sequences to assess visual reasoning capabilities. Results reveal a significant gap: VLMs struggle to match human performance. While human-annotated textual descriptions improve model accuracy, they do not affect human performance, suggesting that models rely more on language priors than on visual understanding. Even with annotations, VLMs fall short of human-level reasoning, underscoring persistent challenges in visual reasoning. A deeper analysis across sub-categories shows that VLMs perform relatively better on videos where temporal and causal reasoning are dominant, compared to those where contextual and spatial reasoning are dominant. They also perform better on everyday tasks than on specialized ones. * * footnotetext: These authors contributed equally to this work 1 Introduction Transformer-based models ( Vaswani 2017 ) initially focused on pre-training with language data alone ( Radford et al. 2018 ; Devlin et al. 2019 ; Raffel et al. 2020 ) . They later evolved to multi-modal pre-training with the introduction of patch-based training ( Dosovitskiy 2020 ) . Since then, vision large language models (VLMs) have advanced rapidly, increasingly matching or even surpassing human performance across various domains, including coding, mathematics, scientific knowledge, and reasoning. For example, benchmarks like ARC-AGI ( Chollet 2019 ) , where models scored 0% in 2019, now report state-of-the-art models achieving scores between 33% and 55.5% ( Chollet et al. 2024 ) . While this remarkable progress in reasoning capabilities is essential for enhancing the utility of current AI systems, our understanding of how well these models reason about purely visual sequences remains incomplete. Unlike textual reasoning tasks, where progress is well-documented, the field lacks benchmarks that rigorously evaluate visual reasoning without heavy reliance on language priors. Figure 1: Video clip ordering accuracy of VLMs across 12 video categories (3,381 videos) in two modalities, compared to the human and random baseline. Alongside advancements in performance, VLMs have become significantly more efficient, enabling them to process long videos. For instance, open-source models like Qwen2-VL ( Wang et al. 2024b ) can understand videos exceeding 20 minutes in length and handle multiple input videos simultaneously. However, despite these capabilities, current benchmarks do not sufficiently probe models’ ability to infer event sequences from purely visual cues. Leveraging this new capability, we propose an intuitive yet challenging benchmark, SPLICE * * * https://huggingface.co/datasets/prokajevo/splice-benchmark . ( S equential P rocessing for L earning and I nference in C hronological E vents), where the task is to order shuffled clips, cut from original videos, based on the events depicted in them. SPLICE fills this gap by leveraging a dataset where the correct ordering of clips requires multiple types of reasoning, such as causal, temporal, spatial, contextual, and general knowledge reasoning. Unlike previous datasets that rely on automatic or unsupervised segmentations, SPLICE is constructed through a rigorous human curation process. We adapt the COIN instructional video dataset ( Tang et al. 2019 ) , which was originally created for video understanding and event localization, by extracting 3,600 videos spanning 180 tasks across 12 domains (e.g., vehicles, gadgets, cooking). In the "vehicles" domain, for instance, tasks may include changing tires, lights, or fuses. COIN provides event-based annotations, which we use to segment each video into distinct clips before shuffling them. This repurposing necessitates careful filtering and validation to ensure that only meaningful and well-defined tasks remain. Consequently, SPLICE eliminates ambiguous or trivial sequences, making it a stronger probe of true visual reasoning. This event-based structure aims to prevent models from relying solely on the first and last frames or other shortcuts, instead requiring deeper reasoning. For instance, a Karate practice video may be divided into three clips: an opening salutation, practicing movements, and a closing salutation. Since the salutation clips are visually identical but occur at different points, the model must rely on other cues such as breathing, sweat levels, spatial positioning, or background actions to determine their order. In this work, we compare the performance of multiple state-of-the-art models that support multi-video input, including Qwen2-VL ( Wang et al. 2024b ) , Gemini-Flash ( Gemini Team 2024 ) , InternVL2.5 ( Chen et al. 2024 ) , and LlavaOnevision ( Li et al. 2024a ) , across three different input settings: videos only, text only, and videos+text. Additionally, we provide human performance benchmarks and compare them with the performance of these VLMs in both the video-only and video+text settings. Our results indicate that VLMs fall significantly behind humans, particularly in the vision-only setting, where there is a substantial performance gap. The main contributions of this paper can be summarized as follows: • We introduce a simple and yet challenging, human-curated benchmark designed to test a model’s ability to reconstruct event sequences from shuffled video clips. The dataset consists of 3,381 human-validated videos, each segmented into multiple clips that must be ordered correctly. • We show that state-of-the-art models struggle with this task, achieving only 23% to 51% accuracy, while humans consistently score around 85%. • We perform a reasoning-type analysis and show that VLMs perform better when the dominant type of reasoning is causal or temporal, as opposed to contextual or spatial. 2 Related Work As this work evaluates VLMs on various aspects of reasoning, we provide a comprehensive review of reasoning evaluations for VLMs, along with tasks similar to our approach, where videos or images were shuffled and reordered. 2.1 Reasoning in VLMs While reasoning in large language models has been extensively explored ( Huang and Chang 2023 ; Plaat et al. 2024 ; Xu et al. 2025 ) , covering aspects such as temporal reasoning ( Chu et al. 2023 ; Tang and Belle 2024 ; Li et al. 2023 ; Maruthi et al. 2022 ) , causal reasoning ( Zhang et al. 2023 ; Hobbhahn et al. 2022 ) , general knowledge reasoning ( Zhang et al. 2024 ; Wu et al. 2024b ) , and spatial reasoning ( Li et al. 2024b ; Hu et al. 2024 ) , vision language models (VLMs) are considered relatively underexplored ( Wang et al. 2024c ) . This is partly due to their novelty, higher computational demands, and the complexity of evaluating their performance. Nevertheless, several studies have investigated different aspects of reasoning in VLMs ( Wu et al. 2024a ; Li et al. 2024c ; Ko et al. 2023 ; Zhang et al. ) . Several benchmarks have also been developed to evaluate VLMs on different reasoning aspects, such as intuitive physics ( Jassim et al. 2023 ; Weihs et al. 2022 ) , mathematics ( Gupta et al. 2024 ; Chen