Every Bit Counts: A Theoretical Study of Precision–Expressivity Tradeoffs in Quantized Transformers

Sayak Chakrabarti, Toniann Pitassi, Josh Alman · 2026

Read Every Bit Counts: A Theoretical Study of Precision–Expressivity Tradeoffs in Quantized Transformers by Sayak Chakrabarti, Toniann Pitassi, Josh Alman (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02707.txt.

Description

Every bit of numerical precision counts, prove Sayak Chakrabarti and 2 co-authors in a 2026 theory study of quantized transformers. Quantization speeds inference by lowering precision, but its effect on what functions a model can even express was poorly mapped. The authors build a fine-grained tradeoff between expressivity and bits.

For every precision level they exhibit an equality-inspired function that a one-layer softmax transformer can compute with that many bits but provably cannot with one bit fewer. The construction explains the expressivity loss practitioners see in practice. Tasks needing equality-like checks, exact match or membership tests, are especially fragile. Dropping a single bit can cross the key threshold. The moral is precise: choose precision based on the length of equality the task must check. Proofs combine explicit finite-precision constructions with communication-complexity lower bounds, making the threshold tight.

Catalog record

CREATORSayak Chakrabarti, Toniann Pitassi, Josh Alman
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-02707

Browse the full library · About the Tower of Babel