A Dual-Path Architecture for Scaling Compute and Capacity in LLMs
Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] · 2026
Read A Dual-Path Architecture for Scaling Compute and Capacity in LLMs by Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30202.txt.
Introduction
Markus Frey and three co-authors propose a dual-path block scaling compute and capacity separately. Looped transformers reuse one shared block, but at fixed compute they hold less capacity than a baseline transformer. Their layer pairs a deep sublayer, re-applied several times, with a wide feed-forward sublayer applied once. Per-token gates combine both axes. Across two budgets, the dual-path model beats matched models on language and downstream tasks with fewer parameters. Its gates show function words trending wide and punctuation trending deep.
Catalog record
| CREATOR | Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 51490 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.30202
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
A Dual-Path Architecture for Scaling Compute and Capacity in LLMs Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] Abstract Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We propose a novel dual-path block that can flexibly scale compute , the number of sequential operations applied to a hidden state, and capacity , the parameters available at a single step. For this we expose both axes as parallel pathways within a single layer: a deep sublayer re-applied K K times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once. Independent per-token gates combine both axes and allow detailed per-token routing analyses. We show that across two FLOP budgets, our dual-path model surpasses iso-FLOP matched models on language modeling and downstream evaluations, while using fewer parameters than the baseline at matched FLOPs. The learned gates are directly interpretable and show systematic per-token allocation with function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep. 1 Introduction Looped (or recursive) transformers re-apply a shared block K times, trading parameter count for sequential compute ( 4 ; 7 ; 19 ; 26 ) . The appeal is parameter efficiency: L layers looped K times reach the effective depth of KL unshared layers at 1/K the parameters, and recent work shows this is enough to recover much of the reasoning performance of a deeper unshared stack ( 19 ; 26 ) . However, at fixed FLOPs, a looped model has strictly less capacity than an unshared stack of comparable compute, and the gap shows up empirically on tasks that depend on stored knowledge ( 6 ; 26 ) . Looping and width scaling therefore sit on two qualitatively different axes of a transformer layer. Looped models use compute to increase the number of sequential operations applied to a hidden state, while width-scaled models increase capacity , the parameters available at a single step. Standard architectures conflate the two, with every token paying the same cost on both. A looped model puts its whole per-layer feed-forward network (FFN) budget on compute and a width-scaled FFN puts it all on capacity. Neither lets a token that needs more sequential refinement get it without also paying for capacity it does not need, or the other way around. Recent work relaxes this one axis at a time. Mixture-of-Experts ( 20 ; 5 ) routes tokens to a subset of similar experts, scaling capacity sub-linearly in compute. Mixture-of-Depths ( 18 ) routes tokens to skip or apply a layer, varying effective depth per position. Mixture-of-Recursions ( 1 ) varies the number of loop iterations per token, and adaptive halting mechanisms in looped models do the same via learned stopping ( 8 ; 2 ; 6 ) . Each of these routes along a single axis, i.e. which experts, how many loop iterations, or whether to apply a layer at all. We propose a transformer layer that exposes the two axes separately within itself. Each dual-path block contains two parallel sublayers: a deep sublayer applied K times with shared parameters (a loop, as above), and a wide sublayer with an enlarged feed-forward dimension applied once. A learned per-token gate combines them. We train the dual-path block across different wide and deep ratios ( \alpha ) and the iso-FLOP controls for each axis separately. We show that: • At both FLOP budgets, the best dual-path configuration beats both single-axis controls on aggregate language-modelling, commonsense, and math evaluations while using fewer parameters than the width-scaled control . • The learned gate that mixes the compute and capacity path does not collapse. Its allocation depends systematically on layer index, part-of-speech, and task: function words and lexical content (verbs, adjectives) trend wide, while punctuation, symbols, and arithmetic tokens trend deep. \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{ffn}} ) \mathbf{h}^{(\ell)} (a) Standard Block \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{deep}} ) \mathbf{h}^{(\ell)} \times K (b) PureLoop \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{wide}} ) \mathbf{h}^{(\ell)} (c) PureWide \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{wide}} ) \mathbf{h}^{(\ell-1)} \mathbf{h}_{\text{wide}} Wide capacity Attn FFN ( d_{\text{deep}} ) \mathbf{h}^{(k-1)} \mathbf{h}_{\text{deep}} \times K Deep compute + \mathbf{h}^{(\ell)}={\color[rgb]{0.2969,0.5508,0.7227}g_{w}}\,\mathbf{h}_{\text{wide}}+{\color[rgb]{0.7813,0.4492,0.2969}g_{d}}\,\mathbf{h}_{\text{deep}} (d) Dual-path Block Per-token allocation learned gates (g_{w},g_{d}) indicate token preference: function words trend wide \cdot arithmetic and symbolic tokens trend deep 1 gate value 0 wide ( g_{w} ) deep ( g_{d} ) Maria pays 15 € plus a tip of \tfrac{1}{5} , the total = 18 € Figure 1: Block architectures. (a) Standard transformer block. (b) PureLoop : a shared block re-applied K times. (c) PureWide : a single block with enlarged FFN. (d) Dual-path block, which runs (c) and (b) in parallel on the same input and combines them via two per-token sigmoid gates g_{w},g_{d} . Bottom panel: schematic of the learned gates across sequence tokens. 2 Related Work Looped and recursive transformers. Re-applying a shared block across depth has been studied as a parameter-efficient route to scaling transformers since the release of Universal Transformers ( 4 ) . The idea has been revived recently in language modelling: looped decoders scale test-time compute by re-applying a shared block ( 7 ; 19 ; 26 ; 12 ) which saves parameters while scaling compute. The cost of the parameter saving is reduced capacity. 26 show that looped models match standard transformers on knowledge manipulation but not on per-parameter memorisation, and 6 report a corresponding empirical pattern downstream: adaptive looping improves mathematical reasoning but leaves commonsense benchmarks largely flat. Both observations show that looping buys compute at the cost of capacity. Per-token compute allocation. A separate line of work allocates compute adaptively at the token level. Mixture-of-Experts ( 20 ; 5 ) routes each token to a small subset of many parallel experts of the same shape, decoupling parameter count from per-token compute. Mixture-of-Depths ( 18 ) routes tokens to skip or apply a layer, varying effective depth per position. Mixture-of-Recursions ( 1 ) extends this to looped stacks by varying the number of shared-block applications per token. All of these route along a single axis (which experts, how many loop iterations, or whether to apply a layer at all). While MoE and our dual-path block both attach a learned router to a transformer layer, they route over different sets of options. In MoE, the router selects among N feed-forward experts that share the same architecture but hold different learned weights; the choice is over which parameters a token sees. In the dual-path block, the router weighs two sublayers that differ in kind: one is a shared-parameter sublayer applied K times, the other is a wider sublayer applied once. The choice is instead over what type of update a token receives. Furthermore, MoE routers are top- k sparse, whereas our gate is dense. Because both paths are always evaluated in our block, we do not skip compute; we re-allocate it. This also means both updates are observed at every token and every layer, making the gate’s value a direct read-out of how the trained model chose to allocate between two options that were both processed.