Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V
Chirag Shinde · 2026
Read Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V by Chirag Shinde (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10791.txt.
Introduction
Shinde proposes two modifications to transformer attention blocks. A nonlinear pre-projection network between layer normalization and query-key-value projections builds features with no position information before positional encoding applies. A content skip routes those features around attention, letting content bypass position-aware steps where helpful. Frozen-probe tests on Pythia 160M and 410M models show the approach leading, with accuracy up 40.6 percent on LAMBADA and perplexity down 39 percent at 160M. Later layers use the content bypass more than early layers, with no key-value cache overhead.
Catalog record
| CREATOR | Chirag Shinde |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 13168 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10791
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Position-Agnostic Pre-Projection for Transformer Attention: Nonlinear Feature Construction and Content Skip Before Q/K/V Chirag Shinde Abstract We propose two complementary modifications to transformer attention blocks. First, a nonlinear pre-projection MLP is inserted between layer norm and Q/K/V projections, constructing richer features in a position-agnostic manner before any positional encoding is applied. Second, a content skip connection routes the pre-projection’s features around the attention mechanism, allowing content information to bypass position-aware attention where beneficial. In frozen-probe experiments on Pythia-160M and 410M, the combined approach achieves the strongest results across methods: +40.6% LAMBADA accuracy and − - 39% perplexity at 160M scale. Learned skip connection weights reveal a consistent pattern across model sizes: later transformer layers activate the content bypass more strongly than earlier layers, suggesting that deeper layers benefit from content information that does not pass through positional attention. All modifications add no K/V cache overhead. 1 Introduction Modern transformer blocks compute attention through a fixed pipeline: layer normalization, linear Q/K/V projections, rotary positional encoding (RoPE) on Q and K, then scaled dot-product attention. This design has two limitations we address: Linear feature bottleneck. The Q/K/V projections are purely linear. If useful attention patterns depend on nonlinear feature combinations, the model must rely on previous layers’ FFNs to have constructed them—requiring multiple layers for what could be a single-layer operation. Mandatory positional filtering. All information reaching the output must pass through position-aware attention. Content information that is relevant regardless of position (semantic type, syntactic category, entity identity) is forced through the same positional routing as position-dependent information. We propose two modifications (Figure 1 ): 1. A pre-projection : a small nonlinear MLP before Q/K/V that constructs richer, position-agnostic features. 2. A content skip : a learned linear projection from the pre-projection output that bypasses attention and adds directly to the layer output. The pre-projection enriches features for attention; the content skip lets those features also bypass attention where the model learns it is beneficial. Both are position-agnostic and add no K/V cache overhead. 2 Method 2.1 Pre-Projection Between layer normalization and Q/K/V projections, we insert: \displaystyle\hat{x} \displaystyle=\text{RMSNorm}(x) (1) \displaystyle\tilde{x} \displaystyle=\hat{x}+W_{\text{down}}\cdot\sigma(W_{\text{up}}\cdot\hat{x}) (2) where W_{\text{up}}\in\mathbb{R}^{d\times ed} , W_{\text{down}}\in\mathbb{R}^{ed\times d} , \sigma is SiLU, and e=1.25 . The residual connection ensures near-identity initialization. 2.2 Content Skip The same enriched features \tilde{x} are projected through a learned linear map W_{\text{skip}}\in\mathbb{R}^{d\times d} and added to the output after attention: \displaystyle Q,K,V \displaystyle=\tilde{x}W_{Q},\;\tilde{x}W_{K},\;\tilde{x}W_{V} (3) attn_out \displaystyle=\text{Attention}(\text{RoPE}(Q),\text{RoPE}(K),V) (4) out \displaystyle=x+\text{attn\_out}+W_{\text{skip}}\cdot\tilde{x} (5) W_{\text{skip}} is initialized near-zero ( \mathcal{N}(0,10^{-4}) ) so the model starts identical to baseline. The skip learns per-layer how much content information should bypass positional attention. No separate mixing parameter (alpha) is needed—the skip projection’s weight magnitude serves this role naturally and trains via standard backpropagation. Key insight : The pre-projection does double duty. It enriches Q/K/V inputs (position-aware path) and provides content features that bypass attention (position-agnostic path). The model learns to route content information through whichever path is more useful at each layer. Standard block Input RMS norm Q, K, V proj. RoPE (Q, K) Attention + RMS norm FFN + Proposed block Input RMS norm Pre-projection Nonlinear. No position. Q, K, V proj. RoPE (Q, K) Attention + RMS norm FFN + W_{\text{skip}} content skip Figure 1: Standard block (left) vs. proposed block (right). The green pre-projection constructs nonlinear features before Q/K/V. The blue content skip routes those features around attention, adding directly to the output. Both paths are position-agnostic; positional encoding enters only at RoPE. 2.3 Parameter Overhead Table 1: Parameter breakdown for the full method (pre-projection + content skip, e=1.25 ). Model Base Pre-proj Skip Total new Overhead Pythia-160M 162M 17.7M 7.1M 24.8M 13.2% Pythia-410M 405M 62.9M 25.2M 88.1M 17.9% Neither the pre-projection nor the content skip adds K/V cache overhead. Both are fixed-cost per-token transformations that occur before and after the attention computation respectively. 3 Experimental Setup 3.1 Models, Data, and Benchmarks We evaluate on Pythia-160M (12 layers) and Pythia-410M (24 layers) ( Biderman et al., 2023 ) , training on WikiText-103 ( Merity et al., 2016 ) . Benchmarks: LAMBADA ( Paperno et al., 2016 ) (discourse comprehension), HellaSwag ( Zellers et al., 2019 ) (commonsense reasoning), ARC-Easy ( Clark et al., 2018 ) (science QA), and WikiText-103 perplexity. 3.2 Protocol All pretrained parameters are frozen. Only injected parameters train (pre-projection, skip, or LoRA). AdamW, cosine schedule, LR 10^{-5} , effective batch 16, bf16, 500 steps. Evaluations on 500-sample subsets. LoRA ( Hu et al., 2022 ) baselines use rank chosen to approximately match parameter counts. 4 Results 4.1 Main Results Table 2: Frozen probe results. Base model frozen throughout. Bold = best per model per metric. Trainable parameter counts vary due to rank granularity (LoRA) and skip projection ( d^{2} per layer). Model Method Params LAMBADA \uparrow HSwag \uparrow ARC-E \uparrow PPL \downarrow Pythia 160M Baseline — 0.128 0.326 0.358 78.4 Pre-proj 17.7M 0.154 0.318 0.342 73.0 LoRA ( r =480) 26.5M 0.126 0.326 0.346 63.8 Pre-proj + LoRA 21.2M 0.162 0.324 0.352 60.8 Pre-proj + skip 24.8M 0.180 0.338 0.348 47.9 Pythia 410M Baseline — 0.466 0.396 0.436 29.2 Pre-proj 62.9M 0.470 0.392 0.452 25.1 LoRA ( r =640) 94.4M 0.488 0.400 0.446 17.7 Pre-proj + LoRA 72.4M 0.470 0.400 0.450 19.2 Pre-proj + skip 88.1M 0.484 0.398 0.432 17.0 Table 2 presents the full comparison. The pre-projection with content skip achieves the strongest overall results: At 160M scale , pre-proj + skip achieves the best LAMBADA (0.180, +40.6% over baseline), the best HellaSwag (0.338, +3.7%), and the best perplexity (47.9, - 38.9% ). This is the only method that improves HellaSwag—all other methods leave it flat or slightly degrade it. At 410M scale , pre-proj + skip achieves the best perplexity (17.0), surpassing even standalone LoRA with 94.4M parameters (17.7). LAMBADA (0.484) exceeds all methods except standalone LoRA (0.488, which uses more parameters). 4.2 Complementarity: Pre-Projection, LoRA, and Content Skip The three techniques address different limitations: • Pre-projection : nonlinear feature construction before attention. • LoRA : linear projection enrichment within attention. • Content skip : content bypass around attention. At 160M scale, LoRA alone decreases LAMBADA below baseline (0.126) while improving perplexity (63.8)—it optimizes surface prediction without improving comprehension. The pre-projection alone improves comprehension (LAMBADA 0.154) but modestly improves perplexity. The content skip combines both strengths: it achieves the best comprehension and perplexity simultaneously. 4.3 Layer-Wise Content Skip Analysis 0 2 4 6 8 10 0.1 0.15 0.2 0.25 0.3 Layer Skip norm Pythia-160M (12 layers) 0 4 8 12 16 20 24 0.2 0.25 0.3 Layer Skip norm Pythia-410M (24 layers) Figure 2: Learned content skip projection weight norms by layer. Both models show the same pattern: later layers activate the content bypass more strongly. Skip weights are initialized at \mathcal{N}(0,10^{-4}) ; the growth is entirely learned. Figure 2 reveals a consistent pattern across both model sizes: the content skip is activated most strongly in the final layers . At 160M, skip norm grows from 0.13 (layers 0–7) to 0.30 (layer 11)—a 2.3 \times increase. At 410M, skip norm grows from 0.19 (layers 0–10) to 0.28 (layer 23)—a 1.5 \times increase. The growth is concentrated in the final 25% of layers at both scales. This pattern has a natural interpretation. Early transformer layers are building