Average Attention Transformers and Arithmetic Circuits

Lena Ehrmuth, Laura Strieker · 2026

Read Average Attention Transformers and Arithmetic Circuits by Lena Ehrmuth, Laura Strieker (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04683.txt.

Description

Lena Ehrmuth and Laura Strieker analyse the computational power of transformer encoders, the attention-based networks behind modern language models, when they are viewed as functions from sequences to sequences. The question is which families of computations such an encoder can carry out.

They show that average hard attention lets a transformer simulate arithmetic circuits that are supplied to it as part of the input. The simulable circuit families have constant depth while using unbounded addition, binary multiplication, and sign gates, and the transformers in their construction use arithmetic circuits in place of the usual feed-forward blocks. The results hold over the reals, the rationals, and any ring between them.

Catalog record

CREATORLena Ehrmuth, Laura Strieker
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE57 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.04683

Browse the full library · About the Tower of Babel