Average Attention Transformers and Arithmetic Circuits
Lena Ehrmuth, Laura Strieker · 2026
Read Average Attention Transformers and Arithmetic Circuits by Lena Ehrmuth, Laura Strieker (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-04683.txt.
Description
Lena Ehrmuth and Laura Strieker analyse the computational power of transformer encoders, the attention-based networks behind modern language models, when they are viewed as functions from sequences to sequences. The question is which families of computations such an encoder can carry out.
They show that average hard attention lets a transformer simulate arithmetic circuits that are supplied to it as part of the input. The simulable circuit families have constant depth while using unbounded addition, binary multiplication, and sign gates, and the transformers in their construction use arithmetic circuits in place of the usual feed-forward blocks. The results hold over the reals, the rationals, and any ring between them.
Catalog record
| CREATOR | Lena Ehrmuth, Laura Strieker |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.04683