Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering

Yugen Sato Meiji University Kanagawa, Japan [email protected] Tomohiro Takagi Meiji University Kanagawa, Japan [email protected] · 2025

Read Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering by Yugen Sato Meiji University Kanagawa, Japan [email protected] Tomohiro Takagi Meiji University Kanagawa, Japan [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22941.txt.

Description

Yugen Sato and Tomohiro Takagi hunt knowledge neurons in multimodal models in 2025. Models that join vision and language can reason over images, but their internals stay opaque. That opacity feeds errors and invented claims. The pair ask where a given piece of knowledge sits inside such a model.

Their two stage filter first compares activations on altered images, then refines with a gradient based method. Tests on a popular caption dataset score the picks by standard text metrics. Activation heatmaps give a visual cross check. Pinpointing knowledge cells could make multimodal models easier to audit and repair.

Catalog record

CREATORYugen Sato Meiji University Kanagawa, Japan [email protected] Tomohiro Takagi Meiji University Kanagawa, Japan [email protected]
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-22941

Browse the full library · About the Tower of Babel