Variational Neurons in Transformers for Language Modeling

Yves Ruffenach · 2026

Read Variational Neurons in Transformers for Language Modeling by Yves Ruffenach (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-28219.txt.

Introduction

Ruffenach puts uncertainty inside the Transformer itself by swapping its feed-forward units for variational neurons. Standard Transformers stay deterministic until the output layer. His units, built on the EVE design, carry local uncertainty through the network. On small next-token tasks, the variational models stayed stable and predicted about as well as plain ones. Their uncertainty signals proved informative too. The results split task quality, useful depth, and stability into separate traits worth tracking.

Catalog record

CREATORYves Ruffenach
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28892
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.28219

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Variational Neurons in Transformers for Language Modeling Yves Ruffenach Abstract Transformers for language modeling usually rely on deterministic internal computation, with uncertainty expressed mainly at the output layer. We introduce variational neurons into Transformer feed-forward computation so that uncertainty becomes part of the internal computation itself. Concretely, we replace deterministic feed-forward units with local variational units based on EVE while preserving the overall Transformer backbone. We evaluate this design in compact next-token language-modeling settings. We compare deterministic and variational variants with both predictive and probabilistic criteria. Alongside negative log-likelihood, perplexity and accuracy, we analyze calibration, conditional variance, mutual information and latent-usage statistics. The resulting picture is clear. Variational neurons integrate stably into Transformers, preserve strong predictive performance and produce informative uncertainty signals. The experiments also show that task quality, useful depth and internal stability are distinct properties. These results establish variational Transformers as a practical form of uncertainty-aware language modeling. They show that Transformers can predict with an explicit internal structure of uncertainty, which supports stronger probabilistic evaluation and a more informative analysis of model behavior. 1 Introduction Transformer language models are strong predictors, but their internal computations are usually deterministic [ 10 , 8 ] . In most current practice, uncertainty is read from the final predictive distribution or recovered through external procedures such as ensembles, dropout, or post-hoc calibration [ 1 , 3 , 7 , 4 ] . This leaves an important question open. Can uncertainty become part of the computation itself, rather than a quantity observed only at the output? This question is especially relevant for language modeling. Next-token prediction is inherently multi-valued and many continuations can be plausible. A model that represents uncertainty only at the last layer can still predict well, but it provides limited access to how uncertainty is formed, propagated and used inside the network. We study a complementary design in which uncertainty is introduced directly into the intermediate computation. Our goal is not only to predict what comes next, but to predict with an explicit internal structure of uncertainty. Our approach builds on EVE, a local variational neuron with a learned posterior, a learned local prior and an explicit control regime for latent activity. We instantiate these neurons inside the feed-forward computation of compact Transformer blocks for next-token language modeling. This preserves the overall Transformer backbone while replacing deterministic intermediate computation with local variational inference. The resulting model combines self-attention with neuron-level stochastic computation, latent-state control and measurable internal uncertainty signals during training. This design gives us a clean setting to study a practical question. Does local variational computation improve model behavior beyond point prediction? To answer this, we compare deterministic and variational Transformers in compact language modeling experiments with frozen GPT-style embeddings and teacher-forcing windows built from prompt–story pairs. Alongside negative log-likelihood, perplexity and accuracy, we track calibration, mutual information, conditional variance, latent activity and related internal diagnostics. This lets us evaluate not only predictive quality, but also how uncertainty is represented and used during computation. Our contributions are threefold. First, we introduce a Transformer language model whose feed-forward computation is carried by local variational neurons. Second, we pair this architecture with an explicit latent-control mechanism that keeps the stochastic path active and numerically stable. Third, we provide an uncertainty-centered evaluation protocol for compact next-token language modeling, with metrics that expose both predictive behavior and internal latent usage. 2 Related Work Transformers, uncertainty and latent variables. Transformers are the standard architecture for language modeling because self-attention supports strong context modeling and efficient parallel training [ 10 ] . Autoregressive models such as GPT-2 established next-token prediction as a simple and effective training paradigm [ 8 ] . We stay in this setting and change not the task, but the internal feed-forward computation. A large literature studies uncertainty in neural networks through Bayesian or approximate Bayesian methods, including variational weight inference, Monte Carlo dropout and deep ensembles [ 1 , 3 , 7 ] . These methods improve calibration and predictive confidence estimation, which is important because modern neural networks are often miscalibrated [ 4 ] . In parallel, latent-variable models for text have long aimed to capture uncertainty, diversity and higher-level structure, while also highlighting the challenge of posterior collapse [ 6 , 2 , 5 ] . Most of this work uses sequence-level or sentence-level latent variables. Our position. Recent work has also explored uncertainty-aware Transformers through Bayesian dropout variants and Transformer-specific approximate inference schemes [ 9 , 11 ] . Our work is complementary but operates at a different level. Rather than attaching uncertainty to weights, recovering it only at the output, or introducing a single global latent code, we place explicit local variational neurons inside Transformer feed-forward blocks. This makes uncertainty part of the forward computation itself and makes latent usage observable throughout the network. 3 Method 3.1 Overview We study language modeling with Transformers whose feed-forward computation uses local variational neurons. Our starting point is EVE, a neuron-level latent-variable compute primitive. Each neuron maintains a local posterior, a local prior and an explicit internal control regime. We instantiate this primitive inside Transformer blocks for next-token prediction. The result is a language model in which uncertainty is part of the internal computation, not only a property of the output distribution. Figure 1 summarizes the architectural change studied in this paper. The Transformer backbone is preserved, while the deterministic feed-forward computation is replaced by a local variational block. Figure 1: Standard and variational Transformer blocks. The overall Transformer backbone is unchanged. Only the feed-forward computation is replaced by a variational block that infers a local latent distribution, samples a latent state and projects it to model space before the residual update. 3.2 Local variational neuron Let x denote the current neuron input and h its local memory state. Following the EVE formulation, each neuron defines a local posterior q_{\phi}(z\mid x,h)=\mathcal{N}\!\big(\mu_{q}(x,h),\operatorname{diag}(\sigma_{q}^{2}(x,h))\big), and a local prior p_{\psi}(z\mid h)=\mathcal{N}\!\big(\mu_{p}(h),\operatorname{diag}(\sigma_{p}^{2}(h))\big). The latent state is sampled through the standard reparameterization z=\mu_{q}+\sigma_{q}\odot\epsilon,\qquad\epsilon\sim\mathcal{N}(0,I), and decoded into an activation y=g_{\theta}(z). This is the key difference with a deterministic neuron. A deterministic neuron emits an activation directly from a hidden transformation. Our neuron emits an activation from a locally inferred latent state. The neuron therefore carries its own probabilistic internal regime. 3.3 Local objective, control, and prior Each neuron contributes a local variational objective that combines a task term, a local KL term, and an internal control term: \mathcal{L}_{\mathrm{local}}=\mathcal{L}_{\mathrm{task}}+\beta\,\mathrm{KL}\!\left(q_{\phi}(z\mid x,h)\,\|\,p_{\psi}(z\mid h)\right)+\mathcal{L}_{\mathrm{control}}. In our setting, \mathcal{L}_{\mathrm{task}} is induced by next-token prediction. The KL term aligns the local posterior with the local prior. The control term keeps the neuron in a useful operating regime. This objective supports both learning and interpretation because it regularizes the latent state while shaping the neuron’s internal behavior. A central control variable is the latent energy \mu^{2}=\frac{1}{d_{z}}\sum_{j=1}^{d_{z}}\mu_{q,j}^{2}, which measures the average squared posterior mean across latent dimensions. Very low latent activity weakens the stochastic path, whereas very high activity destabilizes it. We therefore regulate the neuron around a target latent-energy regime. At a high level, the neuron is encouraged to remain near \mu^{2}_{\mathrm{target}} and inside an admissible band around that target. This yields explicit internal statistics such as inside-band mass, fraction too low, fraction too high, and target gap. In practice, our Transformer implementation

Read the complete text