TIDE: Every Layer Knows the Token Beneath the Context

Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho · 2026

Read TIDE: Every Layer Knows the Token Beneath the Context by Ajay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06216.txt.

Description

Ajay Jaiswal and five co-authors revisit a quiet design choice shared by every modern language model: a token's identity is looked up once, at the input layer, then discarded. That single injection causes two structural failures, the authors argue. Rare tokens get a fraction of the training signal of common ones, and small models can map similar tokens to nearly identical hidden states, a collapse of context.

Their design, TIDE, adds an EmbeddingMemory of independent blocks that map token indices to context free semantic vectors. Those vectors are computed once and injected into every layer through a depth conditioned router with a learnable null option. Theory and experiments support the design. The authors report gains across language modeling and downstream tasks from repairing the single injection weakness.

Catalog record

CREATORAjay Jaiswal, Lauren Hannah, Han-Byul Kim, Duc Hoang, Mehrdad Farajtabar, Minsik Cho
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE72 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06216

Browse the full library · About the Tower of Babel