Hierarchical vs. Flat Iteration in Shared-Weight Transformers Sang-Il Han 1,2 1 Korea University of Technology and Education 2 Biomonster Co.

2026

Read Hierarchical vs. Flat Iteration in Shared-Weight Transformers Sang-Il Han 1,2 1 Korea University of Technology and Education 2 Biomonster Co. (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14442.txt.

Introduction

Han tests whether shared-weight recurrence with hierarchical structure can match independent-layer stacking in a Transformer language model. His HRM-LM swaps independent layers for a recurrent pair at two speeds: a fast module refining at every step and a slow module shrinking every T steps, run with shared parameters. The sharpest, most robust finding comes from a parameter-matched Universal Transformer ablation at 1.2 billion parameters across five runs: a clear empirical gap separates the two approaches. The study frames that gap as the central result.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE115171
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14442

Browse the full library · About the Tower of Babel