SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground?
Haruhiko Murata, Kazuhiro Hotta · 2026
Read SVD-ViT: Does SVD Make Vision Transformers Attend More to the Foreground? by Haruhiko Murata, Kazuhiro Hotta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02765.txt.
Description
Haruhiko Murata and 1 co-author modify vision transformers so they attend to the object instead of the scenery, in research posted to arXiv in 2026. Self-attention operates globally, with no built-in way to tell foreground from background. Transformers can therefore waste capacity on background clutter and lose classification accuracy.
Their model, SVD-ViT, uses singular value decomposition to put foreground features first. Three components extract and combine the singular vectors that carry object information, damping task-irrelevant factors such as background noise. Tests show improved classification accuracy over standard vision transformers. The learned representations say more about the foreground and less contaminated by background. The approach treats a classical linear algebra tool as an attention guide, steering a foundation-scale model toward what human viewers would call the subject of the image.
Catalog record
| CREATOR | Haruhiko Murata, Kazuhiro Hotta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02765