Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers

Zhexiang Li, Haoyu Wang, Yutong Bao, David P. Woodruff · 2025

Read Efficient Attention via Pre-Scoring: Prioritizing Informative Keys in Transformers by Zhexiang Li, Haoyu Wang, Yutong Bao, David P. Woodruff (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11040.txt.

Description

Zhexiang Li, Haoyu Wang, Yutong Bao, and David Woodruff speed up attention in transformer models in 2025. Attention lets each word weigh the others, but long texts make it costly. Fast approximations can skip tokens that matter most, which hurts quality.

Their fix scores keys for global importance before the fast method runs. Only the top keys get full treatment, picked by clustering or a leverage-style score. Added to a known fast method, the score cut error sharply on very long texts at the same budget. The trick also held up in vision models, keeping most accuracy while saving compute.

Catalog record

CREATORZhexiang Li, Haoyu Wang, Yutong Bao, David P. Woodruff
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11040

Browse the full library · About the Tower of Babel