Support Tokens, Stability Margins, and a New Foundation for Robust LLMs Deepak Agarwal, Dhyey Dharmendrakumar Mavani, Suyash Gupta, Karthik Sethuraman, Tejas Dharamsi Abstract Self-attention is usually described as a flexible, content-adaptive way to mix a token with information from its past. We re-interpret causal self-attention transformers, the backbone of modern foundation models, within a probabilistic framework, much like how classical PCA is extended to probabilistic PCA. However, this re-formulation reveals a surprising and deeper structural insight: due to a change-of-variables phenomenon, a barrier constraint emerges on the self-attention parameters. This induces a highly structured geometry on the token space, providing theoretical insights into the dynamics of LLM decoding. This reveals a boundary where attention becomes ill-conditioned, leading to a margin interpretation similar to classical support vector machines. Analogous to support vectors, this naturally gives rise to the concept of support tokens . Furthermore, we show that LLMs define a consistent stochastic process over (infinite) token sequences, providing a rigorous probabilistic framework for sequence modeling. We propose a Bayesian framework and derive a MAP estimation objective that requires only a minimal modification to standard LLM training: the addition of a smooth log-barrier penalty to the usual cross-entropy loss. We demonstrate that this provides more robust models without sacrificing out-of-sample accuracy and that it is straightforward to incorporate in practice. K eywords Large Language Models \cdot Robustness \cdot Causal Attention \cdot Large Margins \cdot Stochastic Processes \cdot Bayesian prior \cdot MAP \cdot Jacobian \cdot Probabilistic Models 1 Introduction Transformers are the default architecture for sequence modeling across language, vision, and beyond ( 25 ; 6 ; 9 ) . The heart of transformers is causal self-attention, which is often described as a content-adaptive weighted average: each token mixes information from its past according to similarity scores. Moving beyond this colloquial description, we ask a formal question: Does causal self-attention admit an explicit probabilistic interpretation, and if so, what does that interpretation imply about the geometry and inductive bias of the model? In this paper, we give a concrete answer. We show that causal self-attention can be interpreted as a probabilistic model on embeddings (treated as latent variables). A key consequence is a barrier constraint on the parameters of self-attention that emerges through this probabilistic formulation; this induces an interesting geometry that leads to key ramifications. It shows that there is a boundary in the latent space where the attention-induced mapping becomes locally ill-conditioned. This defines a margin to degeneracy — a novel quantitative measure we introduce of how far each position is from that unstable boundary. This, in turn, helps us understand which tokens matter the most (Figure 1.1 ). The stability of a sequence is governed by its margin across positions. Tokens whose contexts are closest to degeneracy become the bottlenecks that dominate the barrier signal, directly analogous to support vectors in large-margin classifiers ( 7 ) . Figure 1.1 : Margin-to-degeneracy geometry (SVM-style analogy). The curve denotes an instability boundary : configurations where the attention-induced latent-noise map becomes locally ill-conditioned (small perturbations can lead to ambiguous or disproportionate changes). A sequence traces a trajectory of embedding configurations across token positions (dots). The margin of a token is its distance to this boundary (arrow). In this schematic, token 1 is the support token because it has the smallest margin (it lies closest to instability), so it governs the sequence-level stability margin and dominates the barrier pressure. This mirrors SVMs, where support vectors are the points closest to the decision boundary and therefore determine the margin. (Precise definitions appear in Section 3 .) Moreover, the sign of this stability functional creates both attractive and repulsive coupling behaviors in the attention mechanism: positive values encourage attention to cluster around similar tokens, whereas negative values spread attention across highly dispersed tokens. This sign-dependent attractive/repulsive behavior is reminiscent of classical spin-coupling phenomena in the Ising model ( 15 ) . The “support token” perspective provides an interpretable explanation for where the additional likelihood pressure concentrates and motivates a practical training lever: adding (or stably approximating) the barrier term yields a controllable tradeoff between data fit and geometric conditioning of attention, analogous to classical regularization paths in regression models ( 23 ; 11 ) . Practically, the probabilistic formulation yields an immediate optimization lens. We work in a Bayesian framework ( 4 ; 19 ) where the usual transformer model of tokens conditioned on embeddings is treated as log-likelihood and we compose it with a log-prior based on a probabilistic causal transformer we identify in this paper. Formulating the log-posterior and performing MAP estimation yields the familiar data-fit term alongside a barrier term that enforces geometric stability. This connects causal self-attention to classical constrained estimation ( 5 ) : maximizing the posterior is equivalent to optimizing a squared-error objective (at noise scale \sigma ) under implicit margin constraints that keep the attention mapping well-behaved. Beyond the single-layer self-attention mechanism, we also clarify how depth composes. In particular, we show how deep transformers can be interpreted as a hierarchical probability model across layers, and that the induced token distributions are consistent across sequence lengths, defining a well-posed stochastic process over token sequences. Contributions. 1. A probabilistic interpretation of causal self-attention. We formalize a causal self-attention layer as a conditional probability model over latent embeddings, which in turn induces a joint probability law over token sequences with an exact likelihood. 2. Margin to degeneracy and a stability-inducing log-barrier. We show that the induced likelihood contains an additional term that defines a margin to a critical degeneracy boundary and yields a smooth barrier against locally unstable attention geometries (Theorem 1 ). 3. Optimization view. We show that the log-likelihood can be written as a squared-error objective at noise scale \sigma together with a margin-based barrier term. Thus, our framework reinterprets existing transformers probabilistically rather than replacing them with a different model. 4. A model-implied training penalty and empirical evidence. Using a Bayesian formulation and MAP estimation ( 4 ; 19 ) , we derive a training penalty induced directly by the probabilistic model. This penalty requires no architectural modification and can be added to standard LLM training objectives; in our experiments, it improves robustness of the trained model. 5. Depth as a hierarchy of conditional priors. We characterize how the probabilistic interpretation composes across depth and identify conditions under which the nontrivial volume correction localizes to the first layer under standard transformer conditioning. This explains why a one-layer probabilistic causal prior is already a natural practical choice in the Bayesian setting. Organization. Section 2 introduces our latent-noise view of causal self-attention as a structural (graphical) model over embeddings. Section 3 develops the single-layer theory and derives the margin/log-barrier term induced by token-dependent attention. Section 4 revisits these results from an optimization perspective, connecting the probabilistic objective to a squared-error formulation with a stability constraint, and clarifies the resulting training objective. Section 5 establishes that the induced family of token distributions is consistent across sequence lengths (i.e., defines a well-posed stochastic process), which is the key technical ingredient for learning from datasets of variable-length sequences. Section 6 extends the interpretation to depth, showing how layers compose into a hierarchy of conditional priors and how the correction terms behave under standard conditioning. Section 7 evaluates the resulting penalty empirically. Section 8 surveys related work, and Section 9 discusses implications and future directions. All proofs are deferred to the Appendix. 2 A Latent-Noise View of Causal Self-Attention This section introduces the probabilistic object for causal self-attention that is the core of the paper. The goal is not to modify the transformer architecture, but to make explicit the distribution over the model’s embeddings (hidden states) that a causal self-attention layer implicitly defines when viewed as a generative mechanism. Fix a sequence length L and an embedding dimension d . We write x_{1:L}=(x_{1},\dots,x_{L}),\qquad x_{t}\in\mathbb{R}^{d}, for the sequence of continuous embeddings (hidden states) produced by a transformer ( 25 ; 6 ) . When we need to refer to depth, we write x_{t}^{(\ell)} for the layer- \ell embedding, with x_{t}^{(0)} the input token embedding. We assume, without loss of generality for this discussion, that the embeddings are detrended, as the analysis applies equally to residuals. For clarity, we omit the layer index (\ell) in sections focusing on the single-layer setting. The transformer ultimately is used to define a probability model over discrete tokens via cross-entropy loss, but its computation proceeds through these continuous embeddings. Our focus in this paper is: what probabilistic structure does causal self-attention induce over the embedding sequence x_{1:L} ? We adopt a probabilistic perspective: we treat embeddings as random variables rather than fixed activations. Concretely, we posit that embeddings are generated sequentially from latent noise, \varepsilon_{1:L}=(\varepsilon_{1},\dots,\varepsilon_{L}),\qquad\varepsilon_{t}\in\mathbb{R}^{d}, through a causal transformation. This is the same conceptual step that turns deterministic estimators into probabilistic models (e.g., probabilistic PCA ( 24 ) , see also standard latent-variable modeling treatments ( 4 ; 19 ) ). Throughout the main text we will use an isotropic Gaussian base noise, \varepsilon_{t}\sim\mathcal{N}(0,\sigma^{2}I_{d}), (1) where \sigma>0 controls the noise scale. In the limit \sigma\to 0 , the usual deterministic view is recovered; at finite \sigma , the probabilistic formulation reveals additional geometric structure. 2.1 Causal self-attention as a probabilistic model over embeddings Consider a single self-attention layer. For clarity, we start with a single head and write the usual query/key/value projections q_{t}=W_{Q}x_{t},\qquad k_{s}=W_{K}x_{s},\qquad v_{s}=W_{V}x_{s}. Causal self-attention forms a context summary \mu_{t} by attending to the past. In the strict causal form, ( 25 ; 6 ) \alpha_{ts}(x)\;=\;\frac{\exp\!\left(q_{t}^{\top}k_{s}\right)}{\sum_{r1 result, where scalar variance becomes an attention-weighted covariance matrix. All proofs are deferred to Appendix B . We work with the latent-noise formulation from Section 2 . A single strict-causal self-attention layer defines a context summary \mu_{t}(x) from the past, and we study the induced residual map \varepsilon_{t}\;=\;e_{t}(x)\;\triangleq\;x_{t}-\mu_{t}(x), (6) under a tractable base density on \varepsilon_{1:L} (here \varepsilon_{t}\sim\mathcal{N}(0,\sigma^{2}I_{d}) ). The crucial point is that \mu_{t}(x) depends on x_{t} through the query, so x\mapsto\varepsilon is a token-dependent transformation; its local sensitivity depends on the configuration of the attended context. 3.1 Simple case: the scalar ( d=1 ) We begin with d=1 , so x_{t},v_{s}\in\mathbb{R} . Let \alpha_{ts} denote the strict-causal attention weights, and define the attention-weighted mean and variance of the attended values: \bar{v}_{t}\;\triangleq\;\sum_{s0 ). Under the Gaussian noise model \varepsilon_{t}\sim\mathcal{N}(0,\sigma^{2}) , the change-of-variables formula ( 20 ) yields the induced log-density over embeddings: \log p(x_{1:L})\;=\;-\frac{1}{2\sigma^{2}}\sum_{t=1}^{L}e_{t}(x)^{2}\;+\;\sum_{t=1}^{L}\log\left|\frac{\partial e_{t}}{\partial x_{t}}\right|\;-\;L\log\sigma\;+\;\mathrm{const}. (11) Using Proposition 1 , the additional term becomes \sum_{t=1}^{L}\log\left|1-a\,\mathrm{Var}_{t}(x)\right|\;=\;\sum_{t=1}^{L}\log|m_{t}(x)|. (12) This is the missing term relative to a pure prediction-error objective. Its role is a smooth log-barrier: as any margin approaches the degeneracy boundary m_{t}(x)\to 0 , we have \log|m_{t}(x)|\to-\infty , sharply down-weighting such configurations. Thus, the exact log-probability does not merely fit residuals; it also prefers attention geometries that keep the token-dependent map away from local singularities. 3.4 Theory illustration: positive vs. negative coupling in the scalar case The scalar formula ( 8 ) already shows that the sign of the effective coupling a=W_{K}W_{Q} determines the qualitative behavior of the additional likelihood term. When a>0 , the margin m_{t}(x)=1-a\,\mathrm{Var}_{t}(x) decreases with attention-weighted variance and can approach zero, producing a true log-barrier against degeneracy. When a<0 , the same factor becomes 1+|a|\,\mathrm{Var}_{t}(x) , which stays strictly positive: the singular boundary disappears, and the change-of-variables term instead amplifies configurations with larger dispersion rather than excluding them. Figure 3.2 : Prior density under positive and negative coupling ( d{=}1 , n{=}3 ). (a) Conditional density of x_{3} given x_{1}{=}0 , x_{2}{=}2 . Positive coupling ( a{=}+0.25 , red) lowers and broadens the peak (from \approx 0.38 for the Gaussian baseline to \approx 0.32 ) because the change-of-variables factor 1-a\,\mathrm{Var}_{a}<1 reduces log-density at nonzero dispersion. Negative coupling ( a{=}-0.25 , green) sharpens the density and raises the peak to \approx 0.52 because 1-a\,\mathrm{Var}_{a}>1 amplifies the same region. (b) The diagonal factor 1-a\,\mathrm{Var}_{a} (Eq. 8 ) as a function of x_{3} : for a>0 it decreases toward the degeneracy boundary ( \approx 0.75 here, reaching 0 at \mathrm{Var}_{a}=1/a ), while for a<0 it increases above 1 (to \approx 1.25 here) and remains strictly nondegenerate. (c) Density profiles for a\in\{-0.55,-0.35,0,+0.35,+0.55\} : larger positive coupling progressively flattens the density, whereas larger negative coupling progressively sharpens it. Figure 3.3 : Positive vs. negative coupling regimes ( d{=}1 , n{=}5 ). 4000 random sequences from \mathcal{N}(0,4) evaluated under positive coupling ( a{=}+0.2 , top row) and negative coupling ( a{=}-0.2 , bottom row). (a) For a>0 , the diagonal factor follows 1-a\,\mathrm{Var}_{a} exactly, decreases linearly, and reaches 0 near \mathrm{Var}_{a}\approx 5 ; only 3387/4000 sequences remain valid. (b) For a>0 , log-density decreases with \mathrm{Var}_{a} , and configurations near degeneracy are hard-excluded (red points, log-density =-\infty ). (c) For a<0 , the factor follows 1+|a|\,\mathrm{Var}_{a} , increases linearly, and stays strictly above 1 ; all 4000/4000 sequences remain valid, and \mathrm{Var}_{a} extends to \approx 16 . (d) For a<0 , log-density still decreases overall because the Gaussian residual term dominates, but no sequence is excluded and the change-of-variables term partially offsets that decay (about +\log(4.2)\approx+1.4 nats at \mathrm{Var}_{a}=16 ). Together, the panels show that positive coupling induces a true degeneracy barrier, whereas negative coupling removes the barrier and instead promotes larger dispersion. Figures 3.2 and 3.3 illustrate these two regimes in the simplest scalar setting. We fix x_{1}=0 , x_{2}=2 , sweep x_{3}\in[-4,6] , and compare the induced conditional density under positive, zero, and negative coupling. We also evaluate 4000 random length-5 sequences to visualize the population-level relationship between \mathrm{Var}_{t} , the diagonal factor, and the log-density. Figure 3.2 shows the local effect on the conditional density. For positive coupling ( a=+0.25 ), the change-of-variables factor is less than one whenever \mathrm{Var}_{t}>0 , so it subtracts from the log-density and flattens the peak: the maximum density drops from approximately 0.38 (Gaussian reference) to 0.32 . For negative coupling ( a=-0.25 ), the factor is greater than one, so it adds to the log-density and sharpens the peak to approximately 0.52 . As expected from ( 8 ), the diagonal factor follows the exact linear form 1-a\,\mathrm{Var}_{t} , and increasing |a| makes the deviation from the Gaussian more pronounced. Figure 3.3 shows the corresponding population-level behavior. For positive coupling ( a=+0.2 ), the factor decreases linearly toward zero and 613 out of 4000 sequences (15.3%) are excluded by the degeneracy condition; among the remaining 3387 valid sequences, log-density decreases as \mathrm{Var}_{t} increases. For negative coupling ( a=-0.2 ), all 4000 sequences remain valid, the diagonal factor increases as 1+|a|\,\mathrm{Var}_{t} , and the observed range of \mathrm{Var}_{t} extends to approximately 16 (compared with approximately 5 in the positive-coupling case). The log-density still decreases overall because the Gaussian residual term dominates in the tails, but the change-of-variables term now partially offsets that decay instead of imposing a hard barrier. These illustrations make the sign dependence explicit: positive coupling yields a genuine margin-to-degeneracy and a log-barrier, while negative coupling removes the degeneracy boundary and turns the same term into a dispersion-promoting correction. 3.5 General case ( d>1 ): covariance and a matrix margin We now state the corresponding result in d>1 . Define the attention-weighted mean and covariance of the attended values: \bar{v}_{t}\;\triangleq\;\sum_{s0 , consider \min_{x}\;\;\frac{1}{2\sigma^{2}}\sum_{t=1}^{L}\big(x_{t}-\mu_{t}(x)\big)^{2}\qquad\text{s.t.}\qquad m_{t}(x)\geq\delta\;\;\;\forall t. (22) The log-barrier objective ( 21 ) is the classical interior-point relaxation of ( 22 ) ( 5 ) : it replaces hard constraints m_{t}(x)\geq\delta with a barrier that becomes large as m_{t}(x) approaches 0 . This makes the “stability margin” viewpoint precise: likelihood maximization trades off data fit against maintaining a positive margin away from degeneracy. For d>1 , Theorem 1 gives the diagonal block \frac{\partial e_{t}}{\partial x_{t}}=I_{d}-\Sigma_{t}A,\qquad A=W_{K}^{\top}W_{Q}, (23) with \Sigma_{t} the attention-weighted covariance of attended values. Substituting into ( 19 ) yields \mathcal{L}_{\mathrm{prior},\sigma}(x)=\frac{1}{2\sigma^{2}}\sum_{t=1}^{L}\|x_{t}-\mu_{t}(x)\|_{2}^{2}\;-\;\sum_{t=1}^{L}\log\left|\det\!\left(I_{d}-\Sigma_{t}A\right)\right|\quad(+\;\text{const}). (24) Matrix margin. The degeneracy boundary is characterized by \det(I_{d}-\Sigma_{t}A)=0 , i.e., by configurations where 1 becomes an eigenvalue of \Sigma_{t}A . This suggests a natural “margin to degeneracy” at each position, such as m_{t}(x)\;\triangleq\;\left|\det(I_{d}-\Sigma_{t}A)\right|\qquad\text{or}\qquad m_{t}(x)\;\triangleq\;1-\rho(\Sigma_{t}A), (25) where \rho(\cdot) denotes the spectral radius. Different choices are useful in different analyses; the log-likelihood itself selects the log-determinant form in ( 24 ). 4.1 Integration into the full token model So far we have focused on the embedding prior p_{\sigma}(x_{1:L}) induced by causal attention. To model actual token sequences y_{1:L} , we will combine this prior with the standard transformer token likelihood ( 25 ; 6 ) : p(y_{1:L},x_{1:L})=p(y_{1:L}\mid x_{1:L})\;p_{\sigma}(x_{1:L}). (26) Here p(y\mid x) is exactly what is used in practice: given embeddings (hidden states), the decoder head produces categorical distributions over tokens at each position. The prior p_{\sigma}(x) is the new object made explicit by our latent-noise interpretation. MAP training objective. In the full token model we view the categorical decoder as a likelihood and the embedding prior as a prior. Given a token sequence y_{1:L} , we estimate parameters by maximum a posteriori (MAP): (\hat{\theta},\hat{\phi})\;\in\;\arg\max_{\theta,\phi}\;\log P_{\theta}(y_{1:L}\mid x_{1:L})\;+\;\log p_{\phi,\sigma}(x_{1:L}), (27) equivalently minimizing the negative log-posterior, which is precisely cross-entropy plus the prior-induced stability term (up to constants and weighting). In the next section we (i) specify p(y\mid x) in the standard way, (ii) show how marginalizing the embeddings yields a well-defined sequence likelihood p(y) , and (iii) explain why it is important (for learning across multiple sequences) that p(y) defines a consistent stochastic process (Kolmogorov-style) ( 3 ) . This will let us write down and optimize the augmented log-likelihood over pairs \{(y^{(i)},x^{(i)})\} , in which the margin/barrier term enters as a principled, model-implied regularizer. The key insight is that LLMs can be fit using the same training machinery; the only modification is the addition of the log-prior term, which is straightforward to incorporate and leads to more robust model fitting. Section 5 introduces the full model p(y\mid x)\,p_{\sigma}(x) and its marginal p(y) , and establishes consistency across sequence lengths (Kolmogorov-style), so the model assigns coherent probabilities to variable-length token sequences and datasets ( 3 ) . 5 A Token-Level Stochastic Process Induced by the Embedding Prior So far we have constructed an explicit embedding prior p_{\sigma}(x_{1:n}) induced by causal self-attention. To connect this to real transformer language modeling, we now combine this prior with the usual categorical decoder likelihood P(y\mid x) ( 25 ; 6 ) and study the induced token law \{P(y_{1:n})\}_{n\geq 1} . Our main result is that, under strict causality, these finite-dimensional token distributions are consistent across sequence length and therefore define a bona fide stochastic process over infinite token sequences via standard extension theorems ( 3 ; 17 ) . This gives a rigorous probabilistic foundation for viewing a causal transformer as a single law over variable-length token sequences. Readers primarily interested in the practical modeling consequences may skip the proofs and proceed; all measure-theoretic proofs are deferred to Appendix C . 5.1 From the embedding prior to a token-level law Let V=\{v_{1},\dots,v_{K}\} be a finite vocabulary, equipped with the discrete \sigma -algebra 2^{V} . For each n\geq 1 , the length- n token space is V^{n} , and the infinite token space is the product V^{\infty}=\prod_{t=1}^{\infty}V , endowed with the product \sigma -algebra generated by cylinder sets ( 3 ) . Given latent embeddings x_{1:n} , we use the standard transformer decoder likelihood P(y_{1:n}\mid x_{1:n})\;=\;\prod_{t=1}^{n}P(y_{t}\mid y_{