Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring
Arian Eamaz, Farhang Yeganegi, Mojtaba Soltanalian · 2026
Read Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring by Arian Eamaz, Farhang Yeganegi, Mojtaba Soltanalian (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02853.txt.
Description
Arian Eamaz, Farhang Yeganegi, and Mojtaba Soltanalian propose a way to watch whether a transformer model is actually learning well, layer by layer. Training happens in a rough, nonconvex landscape, and a single loss curve hides which layers are under-trained. The risk is acute for costly language models that are reused in frozen form, where a weak layer silently degrades results.
Their peeling framework tests each layer against reference solutions built from the trained model's own middle outputs. Each layer is locally re-optimized and compared across several output targets, giving an achievable baseline per layer. In tests on decoder-only transformers, these references matched or beat the trained model at points in training, exposing waste the loss curve missed. The check still works under binarized and quantized settings.
Catalog record
| CREATOR | Arian Eamaz, Farhang Yeganegi, Mojtaba Soltanalian |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02853