Training Transformers in Cosine Coefficient Space

Mohamed Amine Bergach · 2026

Read Training Transformers in Cosine Coefficient Space by Mohamed Amine Bergach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-04440.txt.

Description

Bergach replaces each transformer linear layer with one storing a subset of DCT coefficients per weight matrix, rebuilt by inverse DCT each forward pass. A 4-layer, 128-dim transformer trained from scratch on Shakespeare text reaches validation loss near 1.60 at half the trainable parameter count, near baseline. The edge of sparse-coefficient over low-rank forms is qualitative, driven by rank flexibility. The DCT wins for its fast transform, which allows a fused reconstruction kernel. The rebuilt matrix never leaves on-chip memory, so parameter savings become bandwidth savings.

Catalog record

CREATORMohamed Amine Bergach
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18425
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.04440

Browse the full library · About the Tower of Babel