SAS: Simulated Attention Score
Chuanyang Zheng Jiankai Sun Yihang Gao Yuehao Wang Peihao Wang, Jing Xiong Liliang Ren Hao Cheng Janardhan Kulkarni Yelong Shen, Atlas Wang Mac Schwager Anderson Schneider Xiaodong Liu Jianfeng Gao · 2025
Read SAS: Simulated Attention Score by Chuanyang Zheng Jiankai Sun Yihang Gao Yuehao Wang Peihao Wang, Jing Xiong Liliang Ren Hao Cheng Janardhan Kulkarni Yelong Shen, Atlas Wang Mac Schwager Anderson Schneider Xiaodong Liu Jianfeng Gao (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07694.txt.
Description
A research team presents SAS in 2025, a way to fake a larger attention setup inside a Transformer at low cost. Attention scores drive the model, and more heads with wider heads lift scores when size per head stays rich. SAS maps small head codes into a wide space to mimic that gain with no new weights.
The trick extends to key and query codes as well, which adds range at the same model size. A lean merge step caps the cost of the wide play. Tests across sets and tasks show clear gains over past attention forms.
Catalog record
| CREATOR | Chuanyang Zheng Jiankai Sun Yihang Gao Yuehao Wang Peihao Wang, Jing Xiong Liliang Ren Hao Cheng Janardhan Kulkarni Yelong Shen, Atlas Wang Mac Schwager Anderson Schneider Xiaodong Liu Jianfeng Gao |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07694