Training Infinitely Deep and Wide Transformers

Raphaël Barboni, Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré · 2026

Read Training Infinitely Deep and Wide Transformers by Raphaël Barboni, Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-17660.txt.

Introduction

Raphaël Barboni and three colleagues build a rigorous math picture of transformer training as depth and width grow without bound. ResNet training behaves like steering a neural ordinary differential equation, while transformer training steers a neural partial differential equation, since attention couples many token distributions. Their mean-field model tracks token distributions and attention weights across layers. They prove the forward pass is well-posed and derive the Wasserstein gradient of training risk. Gradient flow, they show, reaches global minima when starting loss is small enough, given kernel injectivity.

Catalog record

CREATORRaphaël Barboni, Takashi Furuya, Maarten V. de Hoop, Gabriel Peyré
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE183209
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.17660

Browse the full library · About the Tower of Babel