ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans

Ashkan Shahbazi, Elaheh Akbari, Darian Salehi, Xinran Liu, Navid Naderializadeh, Soheil Kolouri · 2025

Read ESPFormer: Doubly-Stochastic Attention with Expected Sliced Transport Plans by Ashkan Shahbazi, Elaheh Akbari, Darian Salehi, Xinran Liu, Navid Naderializadeh, Soheil Kolouri (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-07962.txt.

Description

Ashkan Shahbazi and five co-authors redesign the attention step inside transformer models. Attention decides how much each word or patch should weigh on the others, but during training it can pile onto a few tokens and starve the rest. Forcing the attention table to be doubly stochastic, with rows and columns each summing to one, keeps the flow balanced. Older ways to enforce that balance iterate a slow normalization routine.

The new mechanism, ESPFormer, gets the same property from sliced optimal transport plans, computed in parallel with no iteration. A soft sorting trick with a temperature knob keeps the whole step trainable. Across image sorting, point cloud sorting, sentiment reading, and machine translation, the balanced attention lifted scores steadily. The authors post their code online for reuse.

Catalog record

CREATORAshkan Shahbazi, Elaheh Akbari, Darian Salehi, Xinran Liu, Navid Naderializadeh, Soheil Kolouri
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-07962

Browse the full library · About the Tower of Babel