Color in Visual-Language Models: CLIP deficiencies

Guillem Arias, Ramon Baldrich, Maria Vanrell · 2025

Read Color in Visual-Language Models: CLIP deficiencies by Guillem Arias, Ramon Baldrich, Maria Vanrell (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-04470.txt.

Description

Guillem Arias and co-authors probed in 2025 how the vision-language model CLIP understands color. CLIP links images and text and underpins much current AI. On synthetic test sets, it labels the colors of simple stimuli correctly in the main. Two flaws stood out in the experiments.

First, the model rarely calls anything white, gray, or black, badly relating those achromatic tones to color itself. Second, it lets text overpower the picture, a bias proven with a Stroop-style test where word and hue conflict. Digging into single neurons, the team found many cells tuned to text, especially deep in the network. Fewer multimodal color neurons carry the true concept. The authors say color encoding in such models needs rework. Their tests offer a template for auditing other senses in AI systems.

Catalog record

CREATORGuillem Arias, Ramon Baldrich, Maria Vanrell
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-04470

Browse the full library · About the Tower of Babel