Scalable-Softmax Is Superior for Attention
Ken M. Nakanishi · 2025
Read Scalable-Softmax Is Superior for Attention by Ken M. Nakanishi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19399.txt.
Description
Ken Nakanishi studies a weak point of large language models in this 2025 paper. These models use a step called Softmax to weigh which words in context matter most. As context grows, those weights flatten out, so key facts get lost. Nakanishi proposes a fix called Scalable-Softmax, or SSMax. It adjusts the step so focus stays sharp at any context size.
Tests in language modeling show faster learning with the new step. Models also did better on long texts and at pulling out key facts. A look at the attention scores confirmed sharper focus on what matters. Models already part-trained could gain much of the benefit by swapping the step in. The change drops into current designs with little fuss.
Catalog record
| CREATOR | Ken M. Nakanishi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19399