Support Tokens, Stability Margins, and a New Foundation for Robust LLMs
Deepak Agarwal, Dhyey Dharmendrakumar Mavani, Suyash Gupta, Karthik Sethuraman, Tejas Dharamsi · 2026
Read Support Tokens, Stability Margins, and a New Foundation for Robust LLMs by Deepak Agarwal, Dhyey Dharmendrakumar Mavani, Suyash Gupta, Karthik Sethuraman, Tejas Dharamsi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-22271.txt.
Description
Self-attention is usually described as a flexible way to mix a token with its past. Deepak Agarwal and four co-authors reinterpret causal transformers inside a probabilistic framework. The move mirrors how classical PCA extends to probabilistic PCA. That reformulation exposes a barrier constraint on attention parameters, and a degeneracy boundary where the attention map turns locally ill-conditioned.
The boundary yields a stability-margin reading like the margin in support vector machines, giving rise to support tokens: the tokens that pin down the boundary. The paper also shows causal transformers define a steady random process over endless token sequences, and derives a Bayesian training goal from the view. Robust language modeling gains a new foundation, with margins to measure.
Catalog record
| CREATOR | Deepak Agarwal, Dhyey Dharmendrakumar Mavani, Suyash Gupta, Karthik Sethuraman, Tejas Dharamsi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 101 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-22271