Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing
Shubham Aggarwal · 2026
Read Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing by Shubham Aggarwal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-27449.txt.
Introduction
Aggarwal argues that the standard Transformer wastes capacity by giving every layer the same number of attention heads. Splitting the hidden dimension equally forces early heads to work in narrow subspaces, too cramped for complex local patterns. The Prism Transformer replaces the uniform layout with a progressive head schedule: head count rises with depth. Early layers use few, wide heads for local composition. In the 2026 machine learning paper, deep layers use many, narrow heads that decompose patterns into specialised features.
The reshuffle is parameter-neutral and compute-neutral, keeping identical weight matrices and operation budgets. Across three model scales, from roughly one hundred million to three quarters of a billion parameters, validation loss drops consistently. Zero-shot benchmark scores improve on commonsense and reasoning suites. Non-uniform allocation unlocks capacity that was already paid for. Architecture, not scale, was the bottleneck. The schedule invites rethinking other uniform design habits.
Catalog record
| CREATOR | Shubham Aggarwal |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.27449
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Prism Transformer: Progressive Head Schedules for Hierarchical Attention Processing Shubham Aggarwal Abstract Multi-head attention conventionally partitions the hidden dimension equally across all heads at every layer, enforcing an identical representational subspace dimension ( d h = d model / h d_{h}=d_{\text{model}}/h ) throughout the model’s depth. In this work, we identify this uniform allocation as a fundamental structural bottleneck: due to their restricted dimensional space, early-layer heads are unable to faithfully capture complex, high-dimensional contextual patterns. To resolve this, we introduce the Prism Transformer, a novel architectural paradigm that replaces the static, uniform head configuration with a progressive head schedule. By monotonically increasing the head count across layers, the Prism Transformer naturally establishes a local-to-global representational hierarchy: early layers leverage fewer, exceptionally wide heads to capture complex, local compositional patterns, while deep layers deploy many, narrow heads to decompose these patterns into specialized linguistic features. Crucially, this structural shift is parameter-neutral, compute-neutral, and introduces zero training or inference overhead, preserving identical weight matrices and FLOP budgets as the standard Transformer. Across three model scales (124M, 354M, and 757M), the Prism Transformer consistently outperforms uniform baselines, achieving consistent reductions in validation loss alongside consistent gains on downstream zero-shot benchmarks (including PIQA, HellaSwag, ARC-Easy, and WinoGrande). Our findings demonstrate that non-uniform subspace allocation unlocks latent capacity within the standard Transformer budget, enabling more effective use of model capacity. 1 Introduction The multi-head attention (MHA) mechanism 18 is the defining component of the Transformer architecture. By projecting queries, keys, and values into h independent subspaces of dimension d_{h}=d_{\text{model}}/h , MHA enables the model to simultaneously attend to multiple distinct representational patterns. This division of the hidden dimension has been held constant across all layers since the original Transformer: every layer in a standard decoder uses the identical number of heads, and therefore operates within the exact same per-head dimension. This uniform allocation embeds a strong, implicit assumption about representation processing: that all layers across a network’s depth benefit equally from the same granularity of attention subspaces. We argue that this assumption introduces a fundamental architectural mismatch between what early layers require and what a uniform schedule forces them to execute. The problem with uniform allocation Early transformer layers are responsible for integrating raw token embeddings into high-level representations that capture complex, local compositional semantic structures. Encoding these intricate local patterns faithfully requires substantial representational capacity per subspace. Under a standard uniform configuration (e.g., h=12 heads over d_{model}=768 ), each early-layer head is restricted to a narrow d_{h}=64 dimensions. Due to this severely restricted dimensional space, individual heads are forced to thinly distribute a small subspace across positions, rendering them physically ill-equipped to resolve complex, high-dimensional local contextual features. As representations progress through the network, their structural needs evolve. Mid-network layers serve as the primary engine for global contextual integration, aggregating these dense local features across long-range sequence dependencies. Finally, late transformer layers shift away from broad integration toward specialized feature decomposition, extracting and isolating specific fine-grained syntactic, semantic, or task-oriented signals. This downstream refinement objective is naturally well-suited to a high density of narrow, focused subspaces operating in parallel. By forcing every stage, including local composition, global integration, and fine-grained refinement, into identical structural configurations, the conventional uniform schedule ignores the natural local-to-global trajectory of the network. The Prism Transformer. We propose to resolve this structural bottleneck by replacing the rigid, uniform configuration with a progressive head schedule: a non-decreasing sequence (h_{1},h_{2},\dots,h_{L}) where h_{l} denotes the head count of layer l . Early layers utilize a small head count, granting each individual head an exceptionally wide subspace ( d_{h}^{(l)}=d_{\text{model}}/h_{l} is large when h_{l} is small). The head count then increases monotonically across the network’s depth, converging to the standard baseline configuration in later layers. The shape of this schedule traces a decreasing trajectory in d_{h} across depth; forming a narrowing prism through which representations flow from local-to-global. This design yields two critical structural properties. First, it is parameter-neutral: because the MHA projection matrices ( W_{Q},W_{K},W_{V},W_{O} ) maintain their standard \mathbb{R}^{d_{\text{model}}\times d_{\text{model}}} shapes, altering the number of slices does not alter the total parameter count. Second, it is compute-neutral: the dominant attention FLOPs remain mathematically invariant to head count. The Prism Transformer thus unlocks latent representational capacity completely for free. Figure 1 illustrates the contrast between a uniform and a Prism schedule. Figure 1 : Head schedule visualization Comparison of attention head allocations across layers. (Left) Standard baseline featuring uniform heads (a flat schedule across all blocks). (Right) The Prism Transformer featuring an increasing head schedule (a progressive staircase growth across blocks). By expanding the head count in deeper layers, the Prism Transformer implicitly creates a monotonically decreasing per-head dimension ( d_{h} ) trajectory. Contributions Our primary contributions are as follows: • Prism Transformer: We propose a progressive head schedule for decoder-only Transformers that establishes a robust local-to-global representational hierarchy at zero parameter or compute overhead. • Consistent Scale-Invariant Gains: Across three model scales (124M, 354M, 757M), the Prism Transformer achieves lower validation loss than the uniform baselines at identical training compute. (Table 2 ). • Mechanistic Attention Analysis: We conduct a detailed per-layer attention distance analysis, empirically demonstrating that the progressive schedule restructures the network’s attention profile. It encourages tight, highly local semantic aggregation in early wide-head layers and shifts broad, global integration to the mid-network layers where the schedule completes (Section 4 ). • Downstream Benchmark Parity and Improvements: Evaluation across zero-shot language benchmarks (PIQA, HellaSwag, ARC-Easy, and WinoGrande) confirms that our progressive schedule preserves or explicitly improves downstream task-level accuracy. (Table 3 , Figure 2 ). • Head schedule design principles: Through systematic structural ablations, we isolate the specific geometric criteria, such as dimension transition smoothness and block consolidation, that govern schedule effectiveness. We formalize these findings into a compact set of transferable design rules that generalize consistently across all evaluated parameter scales (Section 2 ). 2 The Prism Transformer 2.1 Mathematical Formulation of Progressive Head Schedules Let an L -layer Transformer possess a constant model dimension d_{\text{model}} . A progressive head schedule is defined as a non-decreasing integer sequence \mathcal{S}=(h_{1},h_{2},\dots,h_{L}) where h_{l} denotes the allocation of attention heads at layer l , subject to the following structural constraints: (i) Divisibility: h_{l}\mid d_{\text{model}} for all l\in\{1,\dots,L\} (head count divides model dimension), (ii) Monotonicity: h_{1}\leq h_{2}\leq\cdots\leq h_{L} (monotonically non-decreasing), (iii) Boundary Convergence: h_{L}=h_{\text{base}} (final layers match the baseline head count), 2.2 Empirical Design Guidelines While the constraints in Section 2.1 define the space of valid schedules, optimization within this space requires tuning. Through systematic structural ablations on candidate shapes (detailed in Appendix C ), we formalize two vital design principles that dictate schedule effectiveness: (i) Staircase Smoothness: Abrupt, single-step transitions degrade performance. Optimal schedules utilize localized, multi-layer consolidation phases (e.g., maintaining a specific head dimension for at least 2 to 4 consecutive layers) to stabilize representations before altering granularity. (ii) Baseline Preservation Phase: To ensure robust semantic convergence, at least half of the network’s total layers ( L/2 ) must be dedicated to the baseline head count h_{\text{base}} . 2.3 Hardware Alignment & Complexity A primary advantage of the Prism Transformer is its zero-overhead integration into modern compute clusters. Altering head counts does not touch the parameter space: because the projection matrices