Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering
Yugen Sato Meiji University Kanagawa, Japan [email protected] Tomohiro Takagi Meiji University Kanagawa, Japan [email protected] · 2025
Read Identifying Multi-modal Knowledge Neurons in Pretrained Transformers via Two-stage Filtering by Yugen Sato Meiji University Kanagawa, Japan [email protected] Tomohiro Takagi Meiji University Kanagawa, Japan [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-22941.txt.
Description
Yugen Sato and Tomohiro Takagi hunt knowledge neurons in multimodal models in 2025. Models that join vision and language can reason over images, but their internals stay opaque. That opacity feeds errors and invented claims. The pair ask where a given piece of knowledge sits inside such a model.
Their two stage filter first compares activations on altered images, then refines with a gradient based method. Tests on a popular caption dataset score the picks by standard text metrics. Activation heatmaps give a visual cross check. Pinpointing knowledge cells could make multimodal models easier to audit and repair.
Catalog record
| CREATOR | Yugen Sato Meiji University Kanagawa, Japan [email protected] Tomohiro Takagi Meiji University Kanagawa, Japan [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-22941