On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective
Gabriel Mongaras, Eric C. Larson · 2025
Read On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective by Gabriel Mongaras, Eric C. Larson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23632.txt.
Description
Gabriel Mongaras and Eric C. Larson ask in 2025 why one part of modern AI models works so well. That part, called softmax attention, helps a model weigh which words matter most. Its cost grows fast as text gets longer, so cheaper swaps exist, yet they tend to score worse.
The pair rewrite the method in the form of a recurrent network, an older style of model that works step by step. In that form, each piece of the method can be turned off and tested on its own. Seeing which pieces matter most may show why the full method beats its cheaper swaps. Readers get a clear map of a tool at the heart of most large AI systems.
Catalog record
| CREATOR | Gabriel Mongaras, Eric C. Larson |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 39 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23632