FLIP Reasoning Challenge

Andreas Plesner, Turlan Kuzhagaliyev, Roger Wattenhofer · 2025

Read FLIP Reasoning Challenge by Andreas Plesner, Turlan Kuzhagaliyev, Roger Wattenhofer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-12256.txt.

Introduction

Andreas Plesner, Turlan Kuzhagaliyev, and Roger Wattenhofer introduce FLIP, a benchmark that tests reasoning in artificial intelligence using human verification puzzles. Their 2025 paper draws the tasks from the Idena blockchain, where people prove they are human by picking the logically coherent ordering of four images. The puzzles demand sequence sense, visual storytelling, and common sense.

Top vision and language models scored far below people: the best reached about seventy-eight percent where humans hit ninety-five. Describing the images in text first helped models reason better. Combining fifteen models into an ensemble lifted accuracy to about eighty-five percent. The results expose how far multimodal reasoning still has to go.

Catalog record

CREATORAndreas Plesner, Turlan Kuzhagaliyev, Roger Wattenhofer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-12256

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

FLIP Reasoning Challenge Andreas Plesner, Turlan Kuzhagaliyev, Roger Wattenhofer Abstract Over the past years, advances in artificial intelligence (AI) have demonstrated how AI can solve many perception and generation tasks, such as image classification and text writing, yet reasoning remains a challenge. This paper introduces the FLIP dataset, a benchmark for evaluating AI reasoning capabilities based on human verification tasks on the Idena blockchain. FLIP challenges present users with two orderings of 4 images, requiring them to identify the logically coherent one. By emphasizing sequential reasoning, visual storytelling, and common sense, FLIP provides a unique testbed for multimodal AI systems. Our experiments evaluate state-of-the-art models, leveraging both vision-language models (VLMs) and large language models (LLMs). Results reveal that even the best open-sourced and closed-sourced models achieve maximum accuracies of 75.5% and 77.9%, respectively, in zero-shot settings, compared to human performance of 95.3%. Captioning models aid reasoning models by providing text descriptions of images, yielding better results than when using the raw images directly, 69.6% vs. 75.2% for Gemini 1.5 Pro . Combining the predictions from 15 models in an ensemble increases the accuracy to 85.2%. These findings highlight the limitations of existing reasoning models and the need for robust multimodal benchmarks like FLIP. The full codebase and dataset will be available at https://github.com/aplesner/FLIP-Reasoning-Challenge . 1 Introduction The rapid advancements in machine learning (ML) and artificial intelligence (AI) have led to impressive achievements across various domains, such as object detection and natural language processing. Despite the progress, reasoning—the ability to derive logical conclusions and establish relationships between abstract concepts—remains a significant challenge for AI. Reasoning is a hallmark of human intelligence but has proven difficult to evaluate and replicate in machines. For many years, CAPTCHA systems such as Google reCAPTCHA and hCaptcha relied on the flaws of vision models to differentiate humans from bots. However, recent works demonstrate that machine algorithms now achieve near-perfect performance on these tasks, rendering traditional CAPTCHAs ineffective as measures of “humanness.” For instance, Plesner et al. (2024) showed that reCAPTCHAv2 could be solved by machines with a 100% success rate, underlining the need for new benchmarks beyond recognition tasks. Reasoning is a fundamental capability that AI models must master to perform tasks requiring comprehension, logical inference, and contextual understanding. Existing reasoning benchmarks, such as GSM8K Mirzadeh et al. (2024) , CLEVR Johnson et al. (2016) , and MATH Hendrycks et al. (2021) , provide valuable insights into AI reasoning but suffer from several limitations: • Data Contamination: Many reasoning benchmarks inadvertently overlap with training data, leading to inflated performance metrics that do not reflect true reasoning capabilities. • Narrow Scope: These benchmarks often focus on specific domains, such as mathematical or visual reasoning, without capturing the multimodal and contextual complexity of real-world tasks. (a) Left stack (b) Right stack Figure 1: Example of a Flip challenge from the Idena blockchain. The user is given 4 images presented in two different orderings (also referred to as stacks), and the user must select which stack of images tells a meaningful story. In this example, the answer is the right stack with the story of taking flour, mixing it with other ingredients, frying the dough, and then getting pancakes. Since this is a coherent story, then “right” is the correct answer. Thus, there is a pressing need for benchmarks to evaluate reasoning holistically, particularly in multimodal contexts where visual and logical reasoning are required. To address these challenges, we introduce the FLIP dataset , derived from the Idena blockchain. Idena is a decentralized identity verification system that uses human-generated tasks, known as FLIP challenges, to verify users’ “humanness.” In these challenges, participants are presented with two different orderings of images and must determine which ordering forms a coherent story. These tasks emphasize sequential reasoning and visual storytelling, combining the need for logical and multimodal reasoning. Humans create and verify the tasks through consensus voting, ensuring high-quality annotations. The tasks are designed to be easily solved by humans, as indicated by the fact that out of the 84,600 participant answers, 95.3% of them are correct. One of the key features of the FLIP dataset is its transparency and simplicity, which allow researchers to analyze model failures effectively. The task design–selecting the correct ordering–provides clear ground truth and makes it straightforward to diagnose where models may fail. For example, researchers can assess whether errors stem from misunderstanding the visual content, failing to evaluate logical coherence, or both. This contrasts with more abstract benchmarks, where identifying the root causes of failure can be more challenging, such as requiring mathematical domain knowledge. Furthermore, the FLIP dataset favors short and logical stories, requiring models to assess parameters like sequence coherence and narrative simplicity. This is important to test based on the findings by Pascual et al. (2021) showing that LLMs can generate stories with arbitrary sets of words. Experiments with state-of-the-art models show that reasoning through text input, such as captions generated by image captioning models, performs better than reasoning directly from visual inputs. This highlights the limitations of current AI models in processing raw visual data effectively. Meanwhile, the use of caption-based reasoning ensures that models encounter tasks with many variations rather than relying on a single instance; this should reduce the risk of models having previously encountered and memorized the exact FLIP challenges Mirzadeh et al. (2024) . We introduce the FLIP dataset , a novel benchmark designed to evaluate AI reasoning capabilities in the context of visual storytelling and event-sequence comprehension. Our work comprehensively evaluates state-of-the-art models, comparing their performance on FLIP challenges using direct visual inputs versus text-based descriptions from captioning models. While FLIP’s intuitive design facilitates error analysis, our experiments reveal significant limitations in existing reasoning models and techniques, such as exemplars at inference and task reframing, when applied to FLIP challenges. Even the best-performing open-sourced models achieve maximum accuracy of 75.5% in zero-shot settings, compared to 95.3% achieved by human users. These findings underscore the need for robust, multimodal reasoning benchmarks and highlight the gap between current AI capabilities and human-level reasoning. 2 Related Work Image Captioning Image captioning, as a fundamental task in computer vision, has seen significant advancements with the introduction of models like CLIP Radford et al. (2021) , which pioneered the ability to bridge the understanding of text and images through natural language supervision. CLIP aligns visual and textual representations in a shared embedding space, enabling robust performance across various multimodal tasks. BLIP-2 Li et al. (2023) extended this paradigm by integrating frozen image encoders with large language models, improving multimodal understanding. Similarly, Flamingo by ( Alayrac et al., 2022 ) demonstrated strong few-shot learning capabilities by leveraging large-scale pretraining on multimodal datasets. Earlier works, such as Conceptual Captions, laid the foundation by providing large-scale datasets for training captioning models Sharma et al. (2018) ; Suhr et al. (2019) , and improving how models learn from image multi-modal data Cho et al. (2021) . However, despite these advances, captioning models often struggle to generate contextually rich descriptions necessary for reasoning tasks Cornia et al. (2020) . Large Language Models and Vision-Language Models The integration of vision and language has propelled the development of large models capable of multimodal reasoning. GPT-3 ( Brown et al.,

Read the complete text