A Dual-Path Architecture for Scaling Compute and Capacity in LLMs

Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] · 2026

Read A Dual-Path Architecture for Scaling Compute and Capacity in LLMs by Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-30202.txt.

Introduction

Markus Frey and three co-authors propose a dual-path block scaling compute and capacity separately. Looped transformers reuse one shared block, but at fixed compute they hold less capacity than a baseline transformer. Their layer pairs a deep sublayer, re-applied several times, with a wide feed-forward sublayer applied once. Per-token gates combine both axes. Across two budgets, the dual-path model beats matched models on language and downstream tasks with fewer parameters. Its gates show function words trending wide and punctuation trending deep.

Catalog record

CREATORMarkus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51490
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2605.30202

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

A Dual-Path Architecture for Scaling Compute and Capacity in LLMs Markus Frey, Behzad Shomali, Joachim Koehler, Mehdi Ali, Lamarr Institute, Fraunhofer IAIS, University of Bonn [email protected] Abstract Looped transformers apply a shared block multiple times and have emerged as a parameter-efficient route to scaling compute in language models. However, at fixed FLOPs a looped model has strictly less capacity than a baseline transformer. We propose a novel dual-path block that can flexibly scale compute , the number of sequential operations applied to a hidden state, and capacity , the parameters available at a single step. For this we expose both axes as parallel pathways within a single layer: a deep sublayer re-applied K K times with shared parameters, and a wide sublayer with an enlarged feed-forward network applied once. Independent per-token gates combine both axes and allow detailed per-token routing analyses. We show that across two FLOP budgets, our dual-path model surpasses iso-FLOP matched models on language modeling and downstream evaluations, while using fewer parameters than the baseline at matched FLOPs. The learned gates are directly interpretable and show systematic per-token allocation with function words and lexical content trend wide, while punctuation, symbols, and arithmetic tokens trend deep. 1 Introduction Looped (or recursive) transformers re-apply a shared block K times, trading parameter count for sequential compute ( 4 ; 7 ; 19 ; 26 ) . The appeal is parameter efficiency: L layers looped K times reach the effective depth of KL unshared layers at 1/K the parameters, and recent work shows this is enough to recover much of the reasoning performance of a deeper unshared stack ( 19 ; 26 ) . However, at fixed FLOPs, a looped model has strictly less capacity than an unshared stack of comparable compute, and the gap shows up empirically on tasks that depend on stored knowledge ( 6 ; 26 ) . Looping and width scaling therefore sit on two qualitatively different axes of a transformer layer. Looped models use compute to increase the number of sequential operations applied to a hidden state, while width-scaled models increase capacity , the parameters available at a single step. Standard architectures conflate the two, with every token paying the same cost on both. A looped model puts its whole per-layer feed-forward network (FFN) budget on compute and a width-scaled FFN puts it all on capacity. Neither lets a token that needs more sequential refinement get it without also paying for capacity it does not need, or the other way around. Recent work relaxes this one axis at a time. Mixture-of-Experts ( 20 ; 5 ) routes tokens to a subset of similar experts, scaling capacity sub-linearly in compute. Mixture-of-Depths ( 18 ) routes tokens to skip or apply a layer, varying effective depth per position. Mixture-of-Recursions ( 1 ) varies the number of loop iterations per token, and adaptive halting mechanisms in looped models do the same via learned stopping ( 8 ; 2 ; 6 ) . Each of these routes along a single axis, i.e. which experts, how many loop iterations, or whether to apply a layer at all. We propose a transformer layer that exposes the two axes separately within itself. Each dual-path block contains two parallel sublayers: a deep sublayer applied K times with shared parameters (a loop, as above), and a wide sublayer with an enlarged feed-forward dimension applied once. A learned per-token gate combines them. We train the dual-path block across different wide and deep ratios ( \alpha ) and the iso-FLOP controls for each axis separately. We show that: • At both FLOP budgets, the best dual-path configuration beats both single-axis controls on aggregate language-modelling, commonsense, and math evaluations while using fewer parameters than the width-scaled control . • The learned gate that mixes the compute and capacity path does not collapse. Its allocation depends systematically on layer index, part-of-speech, and task: function words and lexical content (verbs, adjectives) trend wide, while punctuation, symbols, and arithmetic tokens trend deep. \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{ffn}} ) \mathbf{h}^{(\ell)} (a) Standard Block \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{deep}} ) \mathbf{h}^{(\ell)} \times K (b) PureLoop \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{wide}} ) \mathbf{h}^{(\ell)} (c) PureWide \mathbf{h}^{(\ell-1)} Attn FFN ( d_{\text{wide}} ) \mathbf{h}^{(\ell-1)} \mathbf{h}_{\text{wide}} Wide capacity Attn FFN ( d_{\text{deep}} ) \mathbf{h}^{(k-1)} \mathbf{h}_{\text{deep}} \times K Deep compute + \mathbf{h}^{(\ell)}={\color[rgb]{0.2969,0.5508,0.7227}g_{w}}\,\mathbf{h}_{\text{wide}}+{\color[rgb]{0.7813,0.4492,0.2969}g_{d}}\,\mathbf{h}_{\text{deep}} (d) Dual-path Block Per-token allocation learned gates (g_{w},g_{d}) indicate token preference: function words trend wide \cdot arithmetic and symbolic tokens trend deep 1 gate value 0 wide ( g_{w} ) deep ( g_{d} ) Maria pays 15 € plus a tip of \tfrac{1}{5} , the total = 18 € Figure 1: Block architectures. (a) Standard transformer block. (b) PureLoop : a shared block re-applied K times. (c) PureWide : a single block with enlarged FFN. (d) Dual-path block, which runs (c) and (b) in parallel on the same input and combines them via two per-token sigmoid gates g_{w},g_{d} . Bottom panel: schematic of the learned gates across sequence tokens. 2 Related Work Looped and recursive transformers. Re-applying a shared block across depth has been studied as a parameter-efficient route to scaling transformers since the release of Universal Transformers ( 4 ) . The idea has been revived recently in language modelling: looped decoders scale test-time compute by re-applying a shared block ( 7 ; 19 ; 26 ; 12 ) which saves parameters while scaling compute. The cost of the parameter saving is reduced capacity. 26 show that looped models match standard transformers on knowledge manipulation but not on per-parameter memorisation, and 6 report a corresponding empirical pattern downstream: adaptive looping improves mathematical reasoning but leaves commonsense benchmarks largely flat. Both observations show that looping buys compute at the cost of capacity. Per-token compute allocation. A separate line of work allocates compute adaptively at the token level. Mixture-of-Experts ( 20 ; 5 ) routes each token to a small subset of many parallel experts of the same shape, decoupling parameter count from per-token compute. Mixture-of-Depths ( 18 ) routes tokens to skip or apply a layer, varying effective depth per position. Mixture-of-Recursions ( 1 ) extends this to looped stacks by varying the number of shared-block applications per token. All of these route along a single axis (which experts, how many loop iterations, or whether to apply a layer at all). While MoE and our dual-path block both attach a learned router to a transformer layer, they route over different sets of options. In MoE, the router selects among N feed-forward experts that share the same architecture but hold different learned weights; the choice is over which parameters a token sees. In the dual-path block, the router weighs two sublayers that differ in kind: one is a shared-parameter sublayer applied K times, the other is a wider sublayer applied once. The choice is instead over what type of update a token receives. Furthermore, MoE routers are top- k sparse, whereas our gate is dense. Because both paths are always evaluated in our block, we do not skip compute; we re-allocate it. This also means both updates are observed at every token and every layer, making the gate’s value a direct read-out of how the trained model chose to allocate between two options that were both processed.

Read the complete text