TAP into the Patch Tokens: Leveraging Vision Foundation Model Features for AI-Generated Image Detection
Ahmed Abdullah, Nikolas Ebert, Oliver Wasenmüller · 2026
Read TAP into the Patch Tokens: Leveraging Vision Foundation Model Features for AI-Generated Image Detection by Ahmed Abdullah, Nikolas Ebert, Oliver Wasenmüller (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26772.txt.
Introduction
Abdullah, Ebert, and Wasenmüller benchmark vision foundation models for detecting AI images, a use little explored beyond CLIP. Across model families, training objectives, resolutions, and scales, they test ready detection of generated and inpainted images. The best model beats the original CLIP by over 12 percent in accuracy. To use modern models fully, they redesign the classifier head with tunable attention pooling, TAP, pooling output tokens into a view. TAP plus the models sets new best results on two hard wild detection benchmarks.
Catalog record
| CREATOR | Ahmed Abdullah, Nikolas Ebert, Oliver Wasenmüller |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33978 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26772
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
TAP into the Patch Tokens: Leveraging Vision Foundation Model Features for AI-Generated Image Detection Ahmed Abdullah, Nikolas Ebert, Oliver Wasenmüller Abstract Recent methods demonstrate that large-scale pretrained models, such as CLIP vision transformers, effectively detect AI-generated images (AIGIs) from unseen generative models when used as feature extractors. Many state-of-the-art methods for AI-generated image detection build upon the original CLIP-ViT to enhance this generalization. Since CLIP’s release, numerous vision foundation models (VFMs) have emerged, incorporating architectural improvements and different training paradigms. Despite these advances, their potential for AIGI detection and AI image forensics remains largely unexplored. In this work, we present a comprehensive benchmark across multiple VFM families, covering diverse pretraining objectives, input resolutions, and model scales. We systematically evaluate their out-of-the-box performance for detecting fully-generated AI-images and AI-inpainted images, and discover that the best model outperforms the original CLIP by more than 12% in accuracy, beating established approaches in the process. To fully leverage the features of a modern VFM, we propose a simple redesign of the classifier head by utilizing tunable attention pooling (TAP), which aggregates output tokens into a refined global representation. Integrating TAP with the latest VFMs yields substantial performance gains across several AIGI detection benchmarks, establishing a new state-of-the-art on two challenging benchmarks for in-the-wild detection of AI-generated and -inpainted images. 1 Introduction Figure 1 : Performance of our approach in terms of individual and mean F1 scores on the OpenSDI benchmark for AI-generated and -inpainted images 51 . Training only on SD1.5 38 , we achieve state-of-the-art performance and generalization on unseen generator models (SD2.1 38 , SD3 11 , SDXL 33 , Flux 21 ), surpassing established methods using only a frozen VFM 4 and our tunable attention pooling (TAP). (a) CLIP-ViT-L/14 35 (b) SIGLIP2-ViT-SO400M-NaFlex/16 46 (c) PE-Core-ViT-G/14 4 Figure 2 : t-SNE analysis of the feature spaces of three pretrained foundational ViT variants: CLIP-ViT-L/14 35 , SIGLIP2-ViT-SO400M-NaFlex/16 46 , and PE-Core-ViT-G/14 4 . Top row shows the cls token output of the frozen visual encoders, while bottom row shows the global-pooled gpl token obtained with our tunable attention pooling (TAP) approach. Features are obtained from the Stable Diffusion 3 test set of OpenSDI 51 Recently, the landscape of both proprietary and open-source AI image generation models has expanded rapidly. The latest generator models 38 ; 21 ; 57 ; 22 continue to push the boundaries of photo-realistic image synthesis. However, this growth in capabilities has enabled malicious actors to misuse AI image generators in a wide range of attacks. For instance, they can be used to spread disinformation on social media, posing a serious threat to both the authenticity of online visual content and individual privacy. In addition to full-image synthesis, AI image generators today are capable of altering the content of a specific region within an input image (i.e. inpainting) with a single text prompt. This capability considerably extends the possible attack vectors for disinformation and fraud beyond full-image synthesis, as only a small fraction of the image may contain generative artifacts or semantic inconsistencies. To counter this growing threat, a variety of methods and benchmarks for AI-generated image (AIGI) detection have been proposed 50 ; 30 ; 44 ; 64 ; 51 ; 53 . A key challenge that recent methods aim to address is the generalization of detectors to unseen generative models. To achieve this, methods are trained to construct a feature space that can maximize generalization while exposed to only a single generator at train-time. Some approaches analyze pixel-level information in the RGB space 30 ; 50 , while others target on detecting low-level artifacts in alternative image representations 12 ; 43 ; 44 . Since their inception, vision transformers (ViTs) 9 have become ubiquitous in computer vision. They are widely used as the backbone for large-scale vision foundation models (VFMs) owing to their ability to scale effectively with both data and model size. Leveraging large-scale pretrained backbones has become a prominent trend in AI-generated image (AIGI) detection, with backbones trained through DINO 31 , Masked Autoencoding 14 , and CLIP 35 being used for their rich semantic features 51 ; 42 ; 30 . Among these, CLIP has emerged as the most adopted model for AIGI detection, due to its unique ability to align visual and textual concepts. Recently, many CLIP-style VFMs have been proposed that leverage improvements to the ViT architecture 41 ; 2 ; 7 . Combined with even larger and more diverse training recipes, the new generation of VFMs demonstrate improved zero-shot classification and transfer to dense tasks 4 ; 46 , significantly outperforming the original CLIP model. Despite this progress, it remains unclear how much these newer VFMs improve performance in AIGI detection. To address this, we present a comprehensive evaluation of modern VFMs for detecting both fully-generated and -inpainted images. Our experiments span multiple model families, pretraining objectives, ViT variants, and input resolutions, systematically addressing this question. In addition, various methods for AIGI detection which employ CLIP for semantic feature extraction, do so by utilizing only a fraction of the full output sequence 30 ; 19 ; 54 , namely, the cls token, while discarding the patch tokens that encode local visual information. This design choice is particularly limiting for AIGI detection, where generative artifacts and inpainted manipulations often appear in localized image regions rather than across the image as a global pattern. As a result, relying solely on the cls token may suppress fine-grained cues that are crucial for detection. To this end, we propose a simple yet effective approach for semantic feature extraction that aligns the full output sequence (patch tokens and the cls token) more closely with the AIGI detection task using a single tunable attention pooling (TAP) layer. Our approach keeps trainable parameters at minimum, while at the same time yielding significant improvements in generalization. Combining TAP features with the latest VFMs yields state-of-the-art detection performance on unseen generators (Figure 1 ). In essence, our contributions are as follows: (i) We carry out a comprehensive benchmark for measuring transferability of out-of-the-box features of modern VFMs to the AIGI detection task, and discover that many VFMs outclass not only CLIP 35 , but also various established detectors 51 ; 44 ; 19 . (ii) We propose an updated design for semantic feature extraction, utilizing both patch tokens and the cls token via tunable attention pooling (TAP). This simple redesign aligns output features closer to the AIGI detection task by enabling the detector to capture local generative artifacts while preserving global semantic patterns. (iii) We establish a new state-of-the-art on two challenging AIGI detection datasets, achieving an improvement of over 29% and 10% in accuracy over prior methods on Chameleon 53 and OpenSDI 51 , respectively. 2 Related Works Figure 3 : Method Overview. Classical semantic feature extraction for AIGI detection relies on the outdated prior of the original CLIP ViT 35 . In addition, many approaches discard the patch tokens and utilize only the cls token. Our approach updates the semantic feature extraction process by swapping the CLIP ViT with an encoder from a vision foundation model (VFM). Moreover, we take advantage of the full output sequence by employing tunable attention pooling (TAP), reducing the dimensionality while also obtaining a