Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency

Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Markos A. Katsoulakis · 2025

Read Optimal Control for Transformer Architectures: Enhancing Generalization, Robustness and Efficiency by Kelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Markos A. Katsoulakis (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13499.txt.

Introduction

Transformers can be viewed as control systems, and a 2025 paper by Kelvin Kan and co-authors uses that view to make them better. The team applies optimal control theory, in a continuous-time form, to both the training and the design of these models.

The method is a plug-in that needs only small changes to existing models, yet it comes with guarantees on how well the model generalizes and resists noise. Seven experiments cover text writing, sentiment reading, image sorting, and point cloud sorting. On a small text model, final test loss fell by 46 percent while the model used 42 percent fewer parameters. Gains held on the larger GPT-2 model as well.

Catalog record

CREATORKelvin Kan, Xingjian Li, Benjamin J. Zhang, Tuhin Sahai, Stanley Osher, Markos A. Katsoulakis
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE140 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-13499

Browse the full library · About the Tower of Babel