We are not able to identify AI-generated images

Adrien Pavão · 2025

Read We are not able to identify AI-generated images by Adrien Pavão (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22236.txt.

Introduction

Adrien Pavão ran a simple experiment with an uncomfortable result: people cannot reliably tell AI-generated images from real photographs. Participants in an interactive web test classified twenty images each, drawn from a set of 120 difficult cases. Real photos came from the CC12M dataset, and the fakes were carefully produced counterparts made with MidJourney.

Across 165 users and 233 sessions, average accuracy was 54 percent, barely above coin-flip guessing, and practice brought only limited improvement. People answered in about seven seconds on average, and some images fooled nearly everyone always. Pavão concludes that human judgment alone is becoming insufficient as synthetic media improves, and calls for greater awareness and ethical guidelines around AI-generated imagery.

Catalog record

CREATORAdrien Pavão
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE10 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-22236

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

We are not able to identify AI-generated images Adrien Pavão Abstract Abstract AI-generated images are now pervasive online, yet many people believe they can easily tell them apart from real photographs. We test this assumption through an interactive web experiment where participants classify 20 images as real or AI-generated. Our dataset contains 120 difficult cases: real images sampled from CC12M, and carefully curated AI-generated counterparts produced with MidJourney. In total, 165 users completed 233 sessions. Their average accuracy was 54%, only slightly above random guessing, with limited improvement across repeated attempts. Response times averaged 7.3 seconds, and some images were consistently more deceptive than others. These results indicate that, even on relatively simple portrait images, humans struggle to reliably detect AI-generated content. As synthetic media continues to improve, human judgment alone is becoming insufficient for distinguishing real from artificial data. These findings highlight the need for greater awareness and ethical guidelines as AI-generated media becomes increasingly indistinguishable from reality. 1 Introduction AI-generated content is becoming a dominant part of the online ecosystem. Images, text, music, and even videos produced by machine-learning models now circulate alongside authentic human-made content, often without being clearly identified as such. We make the hypothesis that most people overestimate their capacity to reliably spot AI-generated images. This confidence rise from a perception bias: individuals mainly remember the cases where they guessed correctly, while the times they were fooled often pass unnoticed. To measure how well humans can truly distinguish real images from AI-generated ones, we collected an images dataset and built an interactive web experiment, accessible at the following URL: https://adrienpavao.com/RealOrAI Participants are shown a sequence of 20 images and must classify each one as either real photo or AI-generated. Their performance allows us to quantify human accuracy, identify the most deceptive images, and analyze how decision time and prior exposure affect the results. Recent studies was performed on similar protocols, on text data ( 2 ) , on images ( 5 ) and mixing audio, images, and text ( 3 ) . The results suggest performance only slightly above random guessing. 2 Data and Methods Our dataset is intentionally small (120 images) but designed to be difficult. This first version acts as a proof of concept before extending the project to larger image sets, as well as other modalities such as music or video. The dataset is balanced, with 60 images depicting women and 60 depicting men. Examples of real and AI-generated images are shown in Figure 1 . Real images were sampled from the CC12M dataset ( 1 ) . We selected random entries whose text description started by either “a man” or “a woman”. Obvious celebrities were removed to avoid easy recognition. Images were then filtered for quality and resized to a consistent resolution. Fake images were generated using MidJourney v7 ( 4 ) . For each selected real image, we extracted its original CC12M text description and image dimension and used it as a prompt seed. We ran each prompt several times with similar parameters (for example: --stylize 50 , --v 7 ) and manually selected the most realistic results. This curation step was essential, as many generated candidates still looked artificial or stylized. We acknowledge that this introduces a selection bias: the AI-generated images in our dataset are not a random sample, but the most convincing ones among several generations. We consider this bias acceptable, and even desirable, for the purpose of this study. Our goal is to evaluate human performance on the hardest cases, not on obviously synthetic images. Moreover, in real online settings, people who share AI-generated portraits or photographs typically select the most realistic outputs rather than posting the first attempt. Our curation process therefore reflects common user behavior. Here are a few examples of prompts used: - “A woman stands in front of a restaurant door and smiles. –ar 3:4 –style raw –v 7 –stylize 50” - “A man walks through an airport with high ceilings and large windows. –ar 3:2 –style raw –v 7 –stylize 50’ - “A woman walks the runway in a striking red and black dress with a wavy pattern, showcasing her profile. –ar 1667:2500 –style raw –v 7 –stylize 50” Figure 1 : Examples from the dataset: three real images (top row) and three AI-generated images (bottom row). Using actual descriptions from the real data as prompts was done in order to ensure a balanced distribution in scenery, postures and clothes between real and fake data. We acknowledge that this design introduces a form of bias, where participants exposed to similar pictures may infer that they are not part of the same set (either real or fake), thus improving their guessing score. The test consists of 20 images, randomly sampled without replacement. They are shown one by one, and the user must answer “Real” or “AI”. The system records each answer and the response time. Results are saved only once the session is complete. Users can then restart the game; unique users are identified so repeated sessions can be linked and avoided as a source of bias in the analysis. The full labeled dataset, as well as the experiment results, can be found here: https://github.com/didayolo/RealOrAI-dataset 3 Results In total, 165 users performed 233 sessions of 20 images. In average, they obtained an accuracy of 53.76% (SD = 14%, SE = 1%), showing very little capacity to distinguish real and AI-generated images. The scores seem to follow a normal distribution, as shown by Figure 2 . We observe a discrepancy in score between the two classes, as the average accuracy on real pictures is slightly better that on artificial ones, 55.76\% and 51.78\% respectively. Users engaging in multiple sessions only obtained a slight improvements in their scores, as shown in Figure 3 . The average accuracy on first session is 53.33%. This is not surprising, as there is a total of 120 images, so we do not except many data repetition on a small number of sessions. Figure 2 : Scores distribution Figure 3 : Users progress, scatter and tendency. Average score obtained when engaging in sessions. On average, participants responded in 7.33 seconds per image. The response-time distributions indicate that fast decisions yield accuracy close to random, whereas accuracy improves steadily with longer viewing times. The curves for correct and incorrect responses intersect around 15 seconds, suggesting that deliberate inspection is required to exceed chance performance (Figure 4 ). Participants with higher final scores also tend to complete sessions more slowly, with a marked increase in average session duration above roughly 60%, presented in Figure 5 . This pattern supports the idea that accurate discrimination relies on slower, more careful analysis rather than rapid intuition. Figure 4 : Response time density for correct and incorrect answers. Figure 5 : Average time for sessions with score \geq x . Error bars show the standard error on the measurements. It is also interesting to note that users’ performance vary across images, ranging from 26% to 87% accuracy. Each image average accuracy, or difficulty, is presented in Figure 6 , as well as example of the least and most deceptive fake and real images. Figure 6 : Performance obtained by the users on each individual

Read the complete text