Training Transformers in Cosine Coefficient Space
Mohamed Amine Bergach · 2026
Read Training Transformers in Cosine Coefficient Space by Mohamed Amine Bergach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04440.txt.
Description
Bergach replaces each transformer linear layer with one storing a subset of DCT coefficients per weight matrix, rebuilt by inverse DCT each forward pass. A 4-layer, 128-dim transformer trained from scratch on Shakespeare text reaches validation loss near 1.60 at half the trainable parameter count, near baseline. The edge of sparse-coefficient over low-rank forms is qualitative, driven by rank flexibility. The DCT wins for its fast transform, which allows a fused reconstruction kernel. The rebuilt matrix never leaves on-chip memory, so parameter savings become bandwidth savings.
Catalog record
| CREATOR | Mohamed Amine Bergach |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 18425 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.04440