Every Bit Counts: A Theoretical Study of Precision–Expressivity Tradeoffs in Quantized Transformers
Sayak Chakrabarti, Toniann Pitassi, Josh Alman · 2026
Read Every Bit Counts: A Theoretical Study of Precision–Expressivity Tradeoffs in Quantized Transformers by Sayak Chakrabarti, Toniann Pitassi, Josh Alman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02707.txt.
Description
Every bit of numerical precision counts, prove Sayak Chakrabarti and 2 co-authors in a 2026 theory study of quantized transformers. Quantization speeds inference by lowering precision, but its effect on what functions a model can even express was poorly mapped. The authors build a fine-grained tradeoff between expressivity and bits.
For every precision level they exhibit an equality-inspired function that a one-layer softmax transformer can compute with that many bits but provably cannot with one bit fewer. The construction explains the expressivity loss practitioners see in practice. Tasks needing equality-like checks, exact match or membership tests, are especially fragile. Dropping a single bit can cross the key threshold. The moral is precise: choose precision based on the length of equality the task must check. Proofs combine explicit finite-precision constructions with communication-complexity lower bounds, making the threshold tight.
Catalog record
| CREATOR | Sayak Chakrabarti, Toniann Pitassi, Josh Alman |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 104 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-02707