Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions
Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, Sauleh Eetemadi · 2024
Read Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions by Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, Sauleh Eetemadi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-08169.txt.
Introduction
Mohammadmostafa Rostamkhani and co-authors test in 2024 how well AI models that read images handle optical illusions. Question answering about pictures has advanced fast, but test sets rarely include illusions, which fool human eyes in known ways and can trip models too. The team built a new task around such images.
They made four data sets that dress classic illusion effects in digits, clothing items, animals, and letters. Leading models were first tested cold, then fine-tuned on the new data. A simple trick helped a lot: softening the image with blur filters before the model sees it. With that step alone, one model beat human scores on the animal set without any retraining. The results expose a gap between human and machine perception. Tuning and the right preprocessing both raise model strength on illusions, the authors find.
Catalog record
| CREATOR | Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, Sauleh Eetemadi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-08169
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Illusory VQA: Benchmarking and Enhancing Multimodal Models on Visual Illusions Mohammadmostafa Rostamkhani, Baktash Ansari, Hoorieh Sabzevari, Farzan Rahmani, Sauleh Eetemadi Abstract In recent years, Visual Question Answering (VQA) has made significant strides, particularly with the advent of multimodal models that integrate vision and language understanding. However, existing VQA datasets often overlook the complexities introduced by image illusions, which pose unique challenges for both human perception and model interpretation. In this study, we introduce a novel task called Illusory VQA, along with four specialized datasets: IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, and IllusionChar. These datasets are designed to evaluate the performance of state-of-the-art multimodal models in recognizing and interpreting visual illusions. We assess the zero-shot performance of various models, fine-tune selected models on our datasets, and propose a simple yet effective solution for illusion detection using Gaussian and blur low-pass filters. We show that this method increases the performance of models significantly and in the case of BLIP-2 on IllusionAnimals without any fine-tuning, it outperforms humans. Our findings highlight the disparity between human and model perception of illusions and demonstrate that fine-tuning and specific preprocessing techniques can significantly enhance model robustness. This work contributes to the development of more human-like visual understanding in multimodal models and suggests future directions for adapting filters using learnable parameters. 1 Introduction In recent years, there has been significant progress in the field of Visual Question Answering (VQA) and multimodal models, which involves answering questions about images using both vision and language understanding. VQA requires models to interpret visual content and comprehend natural language in order to provide accurate answers. Most existing VQA datasets focus on traditional image understanding and do not consider the challenges posed by novel image illusions. (a) Raw image without illusion (b) Image illusion Figure 1: Illustration of illusion in image Figure 2: Examples of answers of GPT-4o and Gemini to raw images, illusory images, and applied filter one. Through the process of image minification, the human eye becomes adept at uncovering the hidden illusions within. (a): An example of IllusionMNIST, (b): An example of IllusionFashionMNIST, (c): An example of IllusionAnimals, (d): An example of IllusionChar, (e): An example of ’No illusion’ class Image illusion refers to the phenomenon where an image initially portrays one thing, yet upon closer examination and a slight adjustment of focus, it reveals an entirely different representation. An example of this can be found in Figure 1 . The phenomenon we are discussing is called "pareidolia," a psychological effect where people perceive familiar patterns or images (such as faces, animals, or objects) in random or unrelated stimuli. In this instance, you initially see one image, but upon closer inspection, your brain detects another image hidden within the scene. Our brains are naturally wired to recognize patterns, which explains why we often identify familiar shapes in clouds, rock formations, or in images like the one presented here. This is also referred to as a "visual illusion" or "perceptual illusion." In this case, the image consists of an arrangement of shapes and colors that your brain interprets in multiple ways—first as a landscape, and then, upon shifting your perception, as the shape of a rabbit. This type of illusion occurs because the brain attempts to make sense of ambiguous or complex visuals by aligning them with familiar patterns. Although the image itself remains unchanged, your perception shifts, giving the impression of seeing something not intentionally designed into the image. Essentially, illusions like this illustrate how our brains actively interpret and construct reality based on visual input. In this work, we use the terms "pareidolia" and "illusion" interchangeably. The inclusion of illusion images in the dataset is important because it opens up possibilities for applications in steganography and potentially bypassing ethical rules. These types of images for multimodal models act as brainteasers for language models. While the image may contain certain elements, one needs to view it from a different perspective to perceive the illusion within it. It is widely recognized that the human perceptual systems are prone to visual illusions, which can be described as “consistent and persistent discrepancies between a physical state of affairs and its representation in consciousness” Day 1984 . In recent years, there has been a growing interest in model-perceived visual illusions, inspired by the fascinating phenomena observed in human perception Gomez-Villa et al. 2019 ; Lonnqvist et al. 2021 ; Hirsch and Tal 2020 ; Sun and Dekel 2019 . These previous studies have primarily focused on vision, exploring how computer vision models can replicate the effects of illusions by examining the internal representations and comparing them to the perceptual shifts experienced by humans. In spite of the emergence of multimodal models such as CLIP Radford et al. 2021 , BLIP Li et al. 2022b , BLIP-2 Li et al. 2023 , Kosmos-2 Peng et al. 2024 , LLaVA Liu et al. 2023b , MiniGPT-V2 Zhu et al. 2024 ; Chen et al. 2023 , Gemini Team et al. 2024a , and GPT-4V(ision) Yang et al. 2023 , a significant disparity remains between the way humans perceive and interpret images and how these models perceive them . We can use these illusory datasets to make VLMs behave more like humans. We suggest that this approach can also enhance their robustness in OCR capabilities. In this study, our main contributions are as follows. 1. Introduction of new task called Illusory VQA 2. Introduction of IllusionMNIST, IllusionFashionMNIST, IllusionAnimals, and IllusionChar Datasets 1 1 1 All code and results are accessible at https://github.com/IllusoryVQA/IllusoryVQA , and the datasets can be found at https://huggingface.co/VQA-Illusion . : We develop these datasets specifically for benchmarking multimodal models. These datasets include separate train and test sets, providing a comprehensive evaluation framework. Some examples from these datasets can be found in Figure 2 . 3. Evaluation of Zero-Shot Performance: We assess the zero-shot performance of state-of-the-art multimodal models on the aforementioned datasets. This evaluation helps us understand how well these models can generalize and interpret the illusions present in the images. 4. Fine-Tuning and Re-evaluation: We further enhance the evaluation by fine-tuning some of the multimodal models on the training set and then re-evaluating their performance on the test set. This process allows us to measure the effectiveness of fine-tuning in improving the models’ ability to detect illusions. 5. Proposal of an Effective Solution for Illusion Detection: Additionally, we propose a simple yet effective solution for detecting illusions in images. This solution aims to enhance the models’ capability to identify and differentiate between real and illusory elements in the datasets. Our approach involves the application of a fixed Gaussian and blur filter to illusory images. You can find the details of the filter we used in the appendix K . 2 Related Work Fan and Zeng 2023 explores illusory contour perception in deep learning models by creating illusory contour datasets through image distortion techniques based on the abutting grating illusion. It also aims to systematically generate illusory contour samples, construct illusory contour versions of datasets like MNIST, and test these illusions in various deep learning models from TorchVision. Williams and Yampolskiy 2018 creates an optical illusion images dataset consisting of 6725 illusion images from different sources and a smaller dataset of 500 hand-picked images. Gomez-Villa et