Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP

Chenyang Zhao, Kun Wang, Janet H. Hsiao, Antoni B. Chan · 2025

Read Grad-ECLIP: Gradient-based Visual and Textual Explanations for CLIP by Chenyang Zhao, Kun Wang, Janet H. Hsiao, Antoni B. Chan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-18816.txt.

Description

Chenyang Zhao and three co-authors build a tool to explain the choices of a leading image-text model in a 2025 paper. The model, CLIP, matches pictures with captions, but users cannot see which parts of a picture or words in a caption drove a match. Past explanation tools for such models lean on attention maps that are too sparse here to help.

Their method works backward from the match score through the model’s layers. It splits the score’s link to inner features and applies channel and space weights, giving clean heat maps over both the image and the text. The maps show which regions and which words pushed the match up or down. Tests, both by eye and by number, rate the maps as faithful guides to the model’s logic.

Catalog record

CREATORChenyang Zhao, Kun Wang, Janet H. Hsiao, Antoni B. Chan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE86 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-18816

Browse the full library · About the Tower of Babel