SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground?

Haruhiko Murata, Kazuhiro Hotta · 2026

Read SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground? by Haruhiko Murata, Kazuhiro Hotta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02765.txt.

Description

Haruhiko Murata and 1 co-author modify vision transformers so they attend to the object instead of the scenery, in research posted to arXiv in 2026. Self-attention operates globally, with no built-in way to tell foreground from background. Transformers can therefore waste capacity on background clutter and lose classification accuracy.

Their model, SVD-ViT, uses singular value decomposition to put foreground features first. Three components extract and combine the singular vectors that carry object information, damping task-irrelevant factors such as background noise. Tests show improved classification accuracy over standard vision transformers. The learned representations say more about the foreground and less contaminated by background. The approach treats a classical linear algebra tool as an attention guide, steering a foundation-scale model toward what human viewers would call the subject of the image.

Catalog record

CREATORHaruhiko Murata, Kazuhiro Hotta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02765

Browse the full library · About the Tower of Babel