Rational Neural Networks have Expressivity Advantages

Maosen Tang, Alex Townsend · 2026

Read Rational Neural Networks have Expressivity Advantages by Maosen Tang, Alex Townsend (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-12390.txt.

Description

Most neural networks use fixed activation functions, from ReLU to sigmoid, but trainable rational activations may do more with fewer parameters. Maosen Tang led this work with one co-author, Alex Townsend, studying networks with low-degree rational activations. They prove such networks can uniformly approximate any standard fixed-activation network on compact domains, with only polyloglog overhead in size for a given error target.

The reverse direction provably needs far more parameters, on the order of log inverse error in the worst case, an exponential gap that holds for full networks, gated activations, and transformer-style nonlinearities. In practice, rational activations slot into standard architectures and training pipelines, matching or beating fixed activations under identical setups and optimisers. The result is a leaner, more expressive activation choice.

Catalog record

CREATORMaosen Tang, Alex Townsend
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE118 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-12390

Browse the full library · About the Tower of Babel