TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes

Anton Lyubinin, [email protected] · 2026

Read TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes by Anton Lyubinin, [email protected] (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-21724.txt.

Introduction

Lyubinin proposes a new way to build manifold-constrained hyper-connections, which mix multiple residual streams in neural networks. Residual streams carry data past layers, and mixing them blends traits from paths. Prior methods forced double stochasticity by Sinkhorn normalization, paid factorial cost for exact rules, or kept mixing on a submanifold. His transportation Birkhoff polytope build gives exactly doubly stochastic matrices with a quadratic count of free parameters. It keeps full expressivity with no normalization loop, and tests on model training show it works well and stays stable.

Catalog record

CREATORAnton Lyubinin, [email protected]
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE59894
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.21724

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

TBP-mHC: full expressivity for manifold-constrained hyper connections through transportation polytopes Anton Lyubinin, [email protected] Abstract Hyper-Connections (HC) improve residual networks by introducing learnable mixing across multiple residual streams, but unconstrained mixing leads to training instability. Manifold-Constrained Hyper-Connections (mHC) address this by enforcing approximate double stochasticity via Sinkhorn normalization, while mHC-lite ensures exact constraints through convex combinations of permutation matrices at the cost of factorial complexity. KromHC reduces this cost using Kronecker-product parameterizations, but restricts the mixing matrices to a structured submanifold of the Birkhoff polytope . We propose Transportation Birkhoff Polytope (TBP) parameterizations and their Recursive variants (RTBP), which construct exactly doubly stochastic mixing matrices with ( n − 1 ) 2 (n-1)^{2} degrees of freedom. Our approach avoids iterative normalization and combinatorial explosion while preserving full expressivity of the Birkhoff polytope. Empirical results on language model pre-training’ demonstrate competitive performance with improved stability and scalability. The code is publicly available at https://github.com/alyubinin/tbp-mHC 1 Introduction Residual connections ( He et al. 2016 ) are a fundamental component of deep neural networks, defined by the update x_{l+1}=x_{l}+F_{l}(x_{l},W_{l}), which preserves an identity mapping across layers and stabilizes gradient propagation. Recent work ( Zhu et al. 2025 ) generalizes this paradigm through Hyper-Connections (HC), which expand a single residual stream into n parallel streams and introduce learnable mixing matrices. A single HC layer is given by X_{l+1}=H_{l}^{\mathrm{res}}X_{l}+(H_{l}^{\mathrm{post}})^{\top}F_{l}\!\left(H_{l}^{\mathrm{pre}}X_{l},W_{l}\right), where X_{l}\in\mathbb{R}^{n\times C} represents the expanded residue stream, H_{l}^{\mathrm{res}}\in\mathbb{R}^{n\times n} is a mixing matrix, and H_{l}^{\mathrm{pre}},H_{l}^{\mathrm{post}}\in\mathbb{R}^{1\times n} aggregate features from n streams into one and map the layer output back onto the n streams respectively. This formulation enables richer feature propagation across residual streams and improves expressivity. However, replacing the identity with a general linear operator removes the structural guarantees that stabilize deep compositions. In particular, repeated products \prod_{i=1}^{L}H_{l+i}^{\mathrm{res}}, that arise when multiple layers are composed, may deviate from identity-like behavior, leading to loss of feature mean preservation and potential gradient instability. To address this problem, ( Xie et al. 2025 ) recent work constrains H_{l}^{\mathrm{res}} to lie in the Birkhoff polytope ( Birkhoff 1946 ) , \mathcal{B}_{n}\;=\;\left\{X\in\mathbb{R}_{\geq 0}^{n\times n}\;\middle|\;X\mathbf{1}=\mathbf{1},\;X^{\top}\mathbf{1}=\mathbf{1}\right\}, i.e., the set of doubly stochastic (DS) matrices. Such matrices preserve feature averages and have spectral norm bounded by 1, ensuring stability under composition. The Manifold-Constrained Hyper-Connections (mHC) framework from ( Xie et al. 2025 ) enforces this constraint via the Sinkhorn–Knopp (SK) algorithm ( Sinkhorn and Knopp 1967 ) , which iteratively normalizes rows and columns: H_{l}^{res}=\mathrm{SK}({\rm exp}({\rm mat}(\alpha_{l}^{res}x_{l}^{\prime}W_{l}^{res}+b_{l}^{res}))). While this procedure converges to a doubly stochastic matrix under mild conditions, practical implementations use a finite number of iterations (e.g., K\approx 20 ), yielding only an approximation. Classical results show that convergence can be arbitrarily slow, and finite-iteration outputs may significantly violate the DS constraints, leading to accumulated errors across depth (see discussion in ( Yang and Gao 2026 ) ). Moreover, efficient deployment requires specialized fused kernels and recomputation strategies, increasing engineering complexity. To eliminate the approximation gap, mHC-lite ( Yang and Gao 2026 ) leverages the Birkhoff-von Neumann theorem ( Birkhoff 1946 ; von Neumann 1953 ) , which states that matrix from \mathcal{B}_{n} can be written as a convex combination of permutation matrices, and thus: H_{l}^{res}=\sum_{k=1}^{K}\alpha_{k}P_{k},\qquad\alpha=\mathrm{Softmax}(\theta),\quad P_{k}\in\mathcal{P}_{n}. This parameterization guarantees exact doubly stochasticity and enables straightforward implementation. However, it suffers from factorial parameter growth, as representing the full polytope may require up to n! permutation matrices, making it quickly impractical with the growth of n . KromHC ( Zhou et al. 2026 ) addresses this scalability issue by exploiting the closure of the Birkhoff polytope under Kronecker products: H^{\mathrm{res}}=U^{(K)}\otimes\cdots\otimes U^{(1)},\qquad U^{(i)}\in\mathcal{B}_{n_{i}}. This construction guarantees exact doubly stochasticity while reducing parameter complexity to O(n^{2}C) . However, it restricts the mixing matrices to structured submanifold of \mathcal{B}_{n} , thus limiting expressivity (for detailed spectral analysis see ( Dandachi and Diggs-Galligan 2026 ) ). While the results of ( Zhou et al. 2026 ) show that in the tests, conducted in the paper, KromHC is competitive, one cannot fully rule out the possibility of such limitation affecting performance in certain scenarios. Another issue is related to the restriction to the structured subset in general. When the optimal transport lies outside this subset, optimization proceeds with a nonzero normal gradient component, which can manifest as late-stage gradient amplification or instability. In our limited trials we have observed two such cases of KromHC behavior (see appendix G.7 ). Our implementation differs from the ( Zhou et al. 2026 ) optimizer split, which may affect smoothness, but the use of Muon optimizer ( Liu et al. 2025b ) would not by itself remove the constrained-manifold issue. Also in ( Zhou et al. 2026 ) Muon is applied only to the main branch, whereas the hyper-connection parameters are still optimized with AdamW ( Loshchilov and Hutter 2019 ) , used in our work. These approaches expose a fundamental trade-off between Exactness, Expressivity, Memory efficiency, and Speed. The starting point of this work is recalling that the Birkhoff polytope is a special case of a transportation polytope, \mathcal{T}(r,c)=\left\{X\in\mathbb{R}_{\geq 0}^{n\times n}\;\middle|\;X\mathbf{1}=r,\;X^{\top}\mathbf{1}=c\right\}, with r=c=\mathbf{1} . Transportation polytopes are fundamental objects in operations research (see, for example, ( Taha 2017 ; Brualdi 2006 ) ). We revisit the problem from a transportation perspective and propose the Transportation Birkhoff Polytope (TBP) parameterization for the whole \mathcal{B}_{n} . Our construction is based on the classical North-West Corner greedy algorithm ( Taha 2017 , Ch. 5.3.1) and gives • Exact doubly stochasticity (no approximation), • Minimal parameterization with (n-1)^{2} degrees of freedom, • Full expressivity, covering all of \mathcal{B}_{n} . TBP is optimal on exactness, expressivity, and memory efficiency, and so the trade-off it makes must be the speed. It originates in the sequential structure of the algorithm, which restricts parallelism. It is a common feature of many optimal transport algorithms, limiting their use in machine learning. To address this, we introduce Recursive TBP (RTBP), which decomposes the problem into block subproblems. For example, partitioning the matrix into four blocks: H_{l}^{res}=\begin{pmatrix}X_{11}&X_{12}\\ X_{21}&X_{22}\\ \end{pmatrix}, we recursively construct each block subject to consistency constraints \sum X_{11}+\sum X_{12}=r_{\text{top}},\quad\sum X_{11}+\sum X_{21}=c_{\text{left}}. This hierarchical decomposition enables partial parallelization, which increases the speed of the algorithm, while preserving exact feasibility. We conduct numerical experiments to validate the effectiveness of TBP and RTBP. We consider several modifications that make training more effective and compare the performance with mHC, mHC-lite, and KromHC. Our work does not consider the later approach of ( Dandachi and Diggs-Galligan 2026 ) . We note that, however, ( Dandachi and Diggs-Galligan 2026 ) does not cover the full Birkhoff polytope and uses a family of subsets, depending on a hyperparameter that, with its growth, produce a dense subset of \mathcal{B}_{n} in the limit. Our results show that (R)TBP is a strong alternative competitor with qualitative advantages. We also discover certain properties of other algorithms that are not covered in their original papers. In summary, our contributions are: • We introduce TBP, an exact and minimal parameterization of the Birkhoff polytope based on transportation polytope decomposition. • We develop RTBP, a recursive formulation that improves computational efficiency via hierarchical parallelization. • We provide theoretical analysis showing that TBP and RTBP achieves full expressivity with optimal parameterization. • We demonstrate empirically that TBP and RTBP achieve competitive performance with mHC, mHC-lite, and

Read the complete text