Static Key Attention in Vision
Zizhao Hu, Xiaolin Zhou, Mohammad Rostami · 2024
Read Static Key Attention in Vision by Zizhao Hu, Xiaolin Zhou, Mohammad Rostami (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07049.txt.
Description
Zizhao Hu, Xiaolin Zhou, and Mohammad Rostami test a simpler form of attention inside vision transformers. Their paper came out in two thousand twenty-four. Vision transformers sort images by weighing how parts of an image relate to each other. The usual design builds both the query and the key afresh for each input. The authors ask whether the key can stay fixed instead.
Their tests show that a fixed key can match or even beat the standard design. Placed inside a Metaformer backbone, the fixed-key block also works well as a middle stage. It blends the strengths of plain convolution and self-attention. The result hints that attention can be cut from two steps to one with no loss.
Catalog record
| CREATOR | Zizhao Hu, Xiaolin Zhou, Mohammad Rostami |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 42 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07049