Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer’s MLP Budget
Peter Balogh · 2026
Read Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer’s MLP Budget by Peter Balogh (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-03459.txt.
Introduction
Peter Balogh asks how much of a transformer's heavy nonlinear machinery is actually needed. Every layer's feed-forward network transforms every token elaborately, yet many of those computations may be nearly linear. His 2026 study tests six models across two architectures and three text collections. A tiny learned gate, a single logistic classifier, decides token by token whether the full nonlinear path is required.
The gate works, but its logic resists simple rules. Whether a token needs nonlinearity cannot be predicted from the token itself: routing lists built on one corpus fail completely on another. The decision is fully contextual, best predicted by a token's contribution to the passing residual stream. Still, the gate routed a large share of computations through cheap linear paths at tiny accuracy cost in one architecture, while another family resisted more. Frozen linear layers even improved one fully retrained model.
Catalog record
| CREATOR | Peter Balogh |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57628 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.03459
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Half the Nonlinearity Is Wasted: Measuring and Reallocating the Transformer’s MLP Budget Peter Balogh Abstract We investigate when transformer MLP nonlinearity is actually necessary. Transformer MLPs perform elaborate nonlinear transformations at every position at every layer. We show that a substantial fraction of these computations can be replaced by a precomputed linear matrix at negligible cost. A gate with d + 1 d{+}1 parameters—a single logistic classifier—decides when. But the gate’s decision defies simple characterization. Through systematic investigation across six models (162M–2.8B parameters), two architectures, three corpora, and 50,000+ tokens, we establish a strong negative result: nonlinearity need cannot be predicted from token identity . Per-token “routing lists” built on one corpus show zero correlation ( r < 0.05 r<0.05 ) when tested on another—even within the same domain. Residual stream clustering finds no clean separation (AUC 0.52–0.61). The routing decision is fully contextual: the same token requires nonlinear computation in some contexts and not others, and the best predictor of this is the contextual contribution to the residual stream—not the token embedding. Despite weak per-instance predictability, the gate exploits a heavily skewed distribution where most MLP computations are near-linear, achieving 25–56% linear routing at < < 1% perplexity cost in GPT-2. In GPT-2 Large, 11 of 36 layers beat baseline with gating, and no layer exceeds 3.7% all-linear cost. However, this success is architecture-dependent: Pythia models show higher linearization costs, though the full 32-layer sweep of Pythia-2.8B reveals one layer (L3) that narrowly beats baseline ( − 0.13 % -0.13\% ). As a proof of concept, we progressively replace middle-layer MLPs with frozen linear matrices: 5 of 24 layers can be linearized at zero cost with minimal fine-tuning. With a full training budget (117.9M tokens), 4 linearized layers yield a 10.2% perplexity improvement over the original model—and a two-phase gated approach pushes this to 17.3% (19.00 PPL), beating a vanilla fine-tuning control and confirming that the nonlinear MLPs at these layers were actively harmful. 1 Introduction It is a truth universally acknowledged, that a neural network in possession of nonlinear activations, must be in want of every one of them. It isn’t. “Half the money I spend on advertising is wasted; the trouble is I don’t know which half.” — attributed to John Wanamaker (c. 1890) The same could be said of nonlinearity in transformer MLPs—except, as we show, we can tell you which half. In GPT-2 Medium, 70% of layers can be fully linearized at under 3% perplexity cost, and 40% of individual MLP computations can be replaced by a single matrix multiply with no measurable degradation. At four layers, removing the nonlinearity actually improves performance. Wanamaker never found his wasted half. We found ours—and it’s more like two-thirds. The multilayer perceptron is the workhorse of the transformer. At every layer, at every token position, a two-layer network with a nonlinear activation function transforms the residual stream. These MLPs dominate both parameter count and compute. The universal assumption—implicit in architecture design, explicit in textbooks—is that this nonlinearity is essential. Remove it, and the network collapses to a linear map that cannot learn the complex functions language requires. We test this assumption directly. For each layer of GPT-2 and GPT-2 Medium, we fit a closed-form linear approximation to the MLP via ridge regression and measure the perplexity cost of wholesale replacement. The results are surprising: for most layers, the cost is negligible. Layers 2 through 15 of GPT-2 Medium can be fully linearized at a cost of 1.6–2.5% perplexity, with most middle layers under 3.5%. The lone exceptions are the first layer (catastrophic) and the last (significant, +12.3 %)—a pattern that replicates exactly across model sizes. But the more surprising finding is what happens when we route selectively. We train a gate—a function that examines each activation and decides whether to use the full MLP or its linear surrogate. The minimal effective gate is a logistic regression with d{+}1 parameters (769 for models with d=768 ): a single hyperplane in activation space. This gate routes a large fraction of activations to the linear path at near-zero perplexity cost. At GPT-2 Medium Layer 6, gated routing actually improves perplexity by 0.06%—the linear approximation regularizes the MLP’s predictions on a substantial minority of inputs. What does the gate learn? This is where the story becomes more interesting—and more honest—than a simple interpretability narrative. Initial analysis suggested the gate separates function words from content words, and indeed a correlation exists between the gate direction and function-word identity ( |r|=0.18 to 0.62 ). But deeper investigation reveals this correlation to be an artifact of within-corpus regularities, not a robust signal. When we decompose the MLP input into token embedding and contextual contribution and train separate gates, the context-only gate matches the full gate within 0.004 AUC at every layer—while the token-identity gate adds essentially nothing. When we build per-token “No-Fly lists” (tokens that consistently need nonlinearity) on one corpus and test them on another, the correlation drops to zero ( r=0.041 within Wikipedia, r=-0.062 cross-domain). Over a quarter of “No-Fly” tokens actually flip to negative delta on new text. The gate works not because it identifies which tokens need nonlinearity, but because it reads the context and exploits a statistical regularity: the distribution of nonlinearity need is heavily skewed, with the vast majority of MLP computations being near-linear. The gate needs only to identify the thin tail of instances where nonlinearity matters—a contextual judgment that cannot be reduced to token identity. Contributions. 1. Quantification of MLP linearity. We provide the first systematic measurement of how much MLP nonlinearity each layer actually uses, across six models spanning two architectures and a 17\times parameter range (162M–2.8B). 2. Minimal adaptive gating. We show that a (d{+}1) -parameter linear classifier can route a substantial fraction of MLP activations to a linear surrogate at negligible cost, and that larger gates provide no additional benefit. 3. Strong negative result on token-based routing. We demonstrate through cross-corpus testing (50K tokens, three corpora) that nonlinearity need cannot be predicted from token identity ( r<0.05 ), and that per-token routing lists do not generalize even within the same domain. 4. Context dominates routing. We decompose MLP inputs into token embedding and contextual contribution and show that context alone is sufficient for the routing decision (within 0.004 AUC of the full gate at every layer). 5. Regularization through linearization. We document that at 4 of 23 GPT-2 Medium layers, the gated linear approximation outperforms the full MLP, suggesting systematic overfitting in MLP computation at those layers. 6. Architecture-dependent linearizability. We show that linearizability varies substantially across architectures: GPT-2 models linearize cheaply while Pythia models show higher costs, though at 2.8B scale one layer narrowly beats baseline ( -0.13\% ). 7. Proof of concept: progressive linearization. We demonstrate that 5 of 24 GPT-2 Medium layers can have their MLPs replaced with frozen linear matrices at zero perplexity cost. With proper fine-tuning on diverse data, 4 linearized layers produce a 10.2% perplexity improvement . A two-phase gated approach extends this to 17.3% (19.00 PPL), beating a vanilla fine-tuning control ( -14.8\% ) and validating non-uniform capacity allocation as an architectural principle. 2 Background A transformer language model processes a sequence of token embeddings through L