On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective

Gabriel Mongaras, Eric C. Larson · 2025

Read On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective by Gabriel Mongaras, Eric C. Larson (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23632.txt.

Description

Gabriel Mongaras and Eric C. Larson ask in 2025 why one part of modern AI models works so well. That part, called softmax attention, helps a model weigh which words matter most. Its cost grows fast as text gets longer, so cheaper swaps exist, yet they tend to score worse.

The pair rewrite the method in the form of a recurrent network, an older style of model that works step by step. In that form, each piece of the method can be turned off and tested on its own. Seeing which pieces matter most may show why the full method beats its cheaper swaps. Readers get a clear map of a tool at the heart of most large AI systems.

Catalog record

CREATORGabriel Mongaras, Eric C. Larson
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-23632

Browse the full library · About the Tower of Babel