Does CLIP perceive art the same way we do? CCS: Computing methodologies Computer vision CCS: Computing methodologies Scene understanding CCS: Computing methodologies Neural networks CCS: Computing methodologies Learning latent representations CCS: Computing methodologies Visual content-based indexin
Andrea Asperti, Leonardo Dessì, Maria Chiara Tonetti, Nico Wu · 2025
Read Does CLIP perceive art the same way we do? CCS: Computing methodologies Computer vision CCS: Computing methodologies Scene understanding CCS: Computing methodologies Neural networks CCS: Computing methodologies Learning latent representations CCS: Computing methodologies Visual content-based indexin by Andrea Asperti, Leonardo Dessì, Maria Chiara Tonetti, Nico Wu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-05229.txt.
Introduction
Andrea Asperti, Leonardo Dessì, Maria Chiara Tonetti, and Nico Wu ask whether the image-text model CLIP sees paintings the way people do, in a 2025 paper. CLIP links pictures and words through joint embeddings and is often used to guide image generation. The authors probed it on human-made and AI-made art across content, scene sense, style, period, and visible artifacts, comparing its answers with human labels and expert benchmarks.
CLIP showed real strengths alongside clear limits, above all around aesthetic cues and artistic intent. The authors argue that using it to steer creative tools demands better interpretability. Nuance and subjectivity, central to art, are exactly where the model's vision diverges from ours.
Catalog record
| CREATOR | Andrea Asperti, Leonardo Dessì, Maria Chiara Tonetti, Nico Wu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-05229
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Does CLIP perceive art the same way we do? CCS: Computing methodologies Computer vision CCS: Computing methodologies Scene understanding CCS: Computing methodologies Neural networks CCS: Computing methodologies Learning latent representations CCS: Computing methodologies Visual content-based indexing and retrieval CCS: Information systems Multimedia content creation CCS: Applied computing Fine arts Andrea Asperti, Leonardo Dessì, Maria Chiara Tonetti, Nico Wu Abstract . CLIP has emerged as a powerful multimodal model capable of connecting images and text through joint embeddings, but to what extent does it “see” the same way humans do—especially when interpreting artworks? In this paper, we investigate CLIP’s ability to extract high-level semantic and stylistic information from paintings, including both human-created and AI-generated imagery. We evaluate its perception across multiple dimensions—content, scene understanding, artistic style, historical period, and the presence of visual deformations or artifacts. By designing targeted probing tasks and comparing CLIP’s responses to human annotations and expert benchmarks, we explore its alignment with human perceptual and contextual understanding. Our findings reveal both strengths and limitations in CLIP’s visual representations, particularly in relation to aesthetic cues and artistic intent. We further discuss the implications of these insights for using CLIP as a guidance mechanism during generative processes, such as style transfer or prompt-based image synthesis. Our work highlights the need for deeper interpretability in multimodal systems, especially when applied to creative domains where nuance and subjectivity play a central role. Figure 1. 3D UMAP projection of image embeddings of National Gallery of Art Dataset extracted from the CLIP ViT-L/14 model. Each point represents a painting from the National Gallery of Art dataset, colored by artistic style. The visualization reveals the semantic organization of the artworks in the latent space, where similar styles tend to cluster together. 1. Introduction In recent years, multimodal models have reshaped the landscape of machine perception and understanding, with CLIP (Contrastive Language–Image Pretraining) ( Radford et al. 2021 ) standing out as one of the most influential. Trained on hundreds of millions of image–text pairs, CLIP has demonstrated remarkable capabilities across a broad range of tasks, including zero-shot classification ( Wang et al. 2025 ; Li et al. 2025 ; Yang et al. 2024 ) , image retrieval ( Yang et al. 2025 ; Baldrati et al. 2024 ; Liu et al. 2024 ) , re-identification ( Li et al. 2023 ; Asperti et al. 2025b ) , prompt-based generation ( Surapaneni and Bhagvati 2024 ; Huang et al. 2023 ) , and semantic search ( Peng et al. 2024 ; Zhou et al. 2024 ) . Its success has made it a foundational component in many state-of-the-art generative models and vision-language pipelines. In particular, CLIP’s ability to align visual and textual modalities has been widely adopted as a guiding mechanism in tasks ranging from text-to-image synthesis (e.g., GLIDE ( Nichol et al. 2022 ) , DALL·E ( Ramesh et al. 2022 ) , Stable Diffusion ( Rombach et al. 2022 ; Esser et al. 2024 ) ) to creative applications such as style transfer and visual storytelling. Yet, despite its ubiquity, the nature and limits of CLIP’s perceptual alignment remain underexplored. While CLIP is optimized to match images with descriptive captions, it is less clear whether this alignment genuinely reflects human perception — especially in complex, subjective, or culturally embedded domains like art. Its capacity to identify what is depicted in an image is impressive, but its understanding of how that content is rendered — including factors such as artistic style, visual coherence, or historical context — is far less understood. Moreover, most evaluations have focused on natural images or benchmark datasets, leaving open the question of how well CLIP performs on ambiguous, interpretive, or aesthetic content, such as paintings — particularly those generated by AI. In this paper, we investigate CLIP’s perceptual capabilities in the domain of visual art, focusing on both human-made and AI-generated paintings. Our goal is to assess how well CLIP captures not only semantic content but also stylistic attributes, temporal signals, and visual deformations. Through a series of probing tasks and analyses, we evaluate CLIP’s representations along multiple interpretive axes, including scene type, artistic style, historical period, and the presence of visual artifacts. To this end, we leverage two richly annotated datasets: classical works from the National Gallery of Art of Washington ( of Art 2024 ) and synthetic paintings from the AI-Pastiche ( Asperti et al. 2025a ) collection. This study serves as a stepping stone toward a broader question: can large vision-language models, such as CLIP, form something akin to an aesthetic sense? Do they internalize representations of style, harmony, or beauty, and if so, are these grounded in visual abstraction, statistical regularities, or biases in their training data? We offer both a conceptual and empirical investigation into these questions. Our findings reveal a consistent gap between CLIP’s textual associations and its ability to perceive visual nuance. While the model performs well on broad semantic alignment, it struggles with stylistic subtleties, the attribution of artistic periods, and the detection of visual defects in generative imagery. These limitations point to a deeper issue: despite its multimodal power, CLIP lacks a robust internal model of aesthetic form. By combining metadata-driven evaluation with perceptual baselines, our work presents a structured critique of CLIP’s performance in aesthetic domains. In doing so, we highlight the need for more interpretable and perceptually grounded multimodal systems, particularly as models like CLIP are increasingly used to guide, evaluate, and curate creative content. As these systems become embedded in artistic workflows and cultural interfaces, it becomes crucial to ask not just what they can recognize or generate but how they see — and whether that way of seeing aligns with our own. 2. Related Works Several recent works have focused on the problem of ascertaining the potential and limitations of CLIP’s vision mechanism and the extent to which the image embedding can be retrieved and effectively exploited through textual prompts. Concerns about CLIP’s capacity to deeply understand the meaning of its textual prompts were raised in ( Pan et al. 2022 ) . In ( Abbasi et al. 2025 ) , a comprehensive analysis of CLIP’s performance limitations in multi-object contexts through controlled experiments is presented. The findings reveal significant biases in both encoders: the image encoder favors larger objects, while the text encoder prioritizes objects mentioned first in descriptions. Several works have been devoted to assess the robustness of CLIP. In ( Rosa et al. 2024 ) , the authors investigate the adversarial robustness of CLIP-based methods in detecting AI-generated images, revealing their vulnerability to white-box attacks. A more comprehensive assessment of CLIP’s robustness along different perspectives is done in ( Tu et al. 2024 ) : despite several limitations, CLIP models exhibit superior robustness to visual factor-level variations compared to more traditional ImageNet models. In ( Lytvyn et al. 2024 ) , the authors present a CLIP-based method for image similarity evaluation that relies on textual descriptions rather than raw visual features. Using cosine similarity between encoded text and image vectors, the approach captures semantic relationships more effectively than traditional metrics, such as SSIM and FSIM, especially in complex images. The results confirm CLIP’s robustness in tasks such as semantic search and classification, despite