TIDE: Every Layer Knows the Token Beneath the Context
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho · 2026
Read TIDE: Every Layer Knows the Token Beneath the Context by Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06216.txt.
Description
Ajay Jaiswal and five co-authors revisit a quiet design choice shared by every modern language model: a token's identity is looked up once, at the input layer, then discarded. That single injection causes two structural failures, the authors argue. Rare tokens get a fraction of the training signal of common ones, and small models can map similar tokens to nearly identical hidden states, a collapse of context.
Their design, TIDE, adds an EmbeddingMemory of independent blocks that map token indices to context free semantic vectors. Those vectors are computed once and injected into every layer through a depth conditioned router with a learnable null option. Theory and experiments support the design. The authors report gains across language modeling and downstream tasks from repairing the single injection weakness.
Catalog record
| CREATOR | Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06216