Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers
Zhexiang Li, Haoyu Wang, Yutong Bao, David P. Woodruff · 2025
Read Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers by Zhexiang Li, Haoyu Wang, Yutong Bao, David P. Woodruff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11040.txt.
Description
Zhexiang Li, Haoyu Wang, Yutong Bao, and David Woodruff speed up attention in transformer models in 2025. Attention lets each word weigh the others, but long texts make it costly. Fast approximations can skip tokens that matter most, which hurts quality.
Their fix scores keys for global importance before the fast method runs. Only the top keys get full treatment, picked by clustering or a leverage-style score. Added to a known fast method, the score cut error sharply on very long texts at the same budget. The trick also held up in vision models, keeping most accuracy while saving compute.
Catalog record
| CREATOR | Zhexiang Li, Haoyu Wang, Yutong Bao, David P. Woodruff |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-11040