Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis
Anzhe Cheng Chenzhong Yin Mingxi Cheng Shukai Duan, Shahin Nazarian Paul Bogdan · 2025
Read Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis by Anzhe Cheng Chenzhong Yin Mingxi Cheng Shukai Duan, Shahin Nazarian Paul Bogdan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01219.txt.
Introduction
Anzhe Cheng and co-authors present a neural network design that forces each layer's weights into a shared, learned set of perpendicular directions, in a paper from 2025. Standard training often leaves weights in a tangled state that dulls features. Their design builds order in by construction rather than by penalty terms.
Paired with normal training, the design beat leading methods on large image classification tests. Its features also set a new mark on matching images with text. A local learning variant skips end to end backpropagation, trains more than twice as fast through parallelism, and still scored higher than the classic method. The authors present the design as a new base pattern for vision networks.
Catalog record
| CREATOR | Anzhe Cheng Chenzhong Yin Mingxi Cheng Shukai Duan, Shahin Nazarian Paul Bogdan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 49 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01219
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Eigen Neural Network: Unlocking Generalizable Vision with Eigenbasis Anzhe Cheng Chenzhong Yin Mingxi Cheng Shukai Duan, Shahin Nazarian Paul Bogdan Abstract The remarkable success of Deep Neural Networks(DNN) is driven by gradient-based optimization, yet this process is often undermined by its tendency to produce disordered weight structures, which harms feature clarity and degrades learning dynamics. To address this fundamental representational flaw, we introduced the Eigen Neural Network (ENN), a novel architecture that reparameterizes each layer’s weights in a layer-shared, learned orthonormal eigenbasis. This design enforces decorrelated, well-aligned weight dynamics axiomatically, rather than through regularization, leading to more structured and discriminative feature representations. When integrated with standard BP, ENN consistently outperforms state-of-the-art methods on large-scale image classification benchmarks, including ImageNet, and its superior representations generalize to set a new benchmark in cross-modal image-text retrieval. Furthermore, ENN’s principled structure enables a highly efficient, backpropagation-free(BP-free) local learning variant, ENN- ℓ \ell . This variant not only resolves BP’s procedural bottlenecks to achieve over 2 × \times training speedup via parallelism, but also, remarkably, surpasses the accuracy of end-to-end backpropagation. ENN thus presents a new architectural paradigm that directly remedies the representational deficiencies of BP, leading to enhanced performance and enabling a more efficient, parallelizable training regime. Introduction Backpropagation(BP), while foundational to deep learning, suffers from fundamental limitations that compromise both training efficiency and biological plausibility. The BP training process frequently leads to misaligned weight structures that harm feature clarity and learning dynamics ( Szegedy et al. 2013 ; Lillicrap et al. 2020 ) . A central limitation of backpropagation is that its weight updates are largely unstructured, yielding highly redundant parameterizations: in standard MLPs and convnets, over 95% of weights can be accurately predicted from a small subset without harming accuracy ( Denil et al. 2013 ) . More recent studies confirm this dynamic. Forward and backward weights trained with BP often remain disordered, leading to ill-conditioned dynamics and slower convergence ( Flügel et al. 2024 ; Ji-An and Benna 2024 ) , and even architectures with random feedback require soft alignment at initialization for stable convergence ( Cheon, Bae, and Paik 2025 ) . For example, Frozen Backpropagation ( Goupy, Tirilly, and Bilasco 2025 ) reports that a ResNet-50 trained on ImageNet retains a forward–feedback cosine alignment below 0.2 after 50 epochs, stalling optimization despite periodic re-synchronization of feedback weights. These findings indicate that standard BP produces feature maps dominated by redundant or entangled directions, especially in deep networks. Beyond weight disorder, BP also enforces strict layer-wise dependencies. Each layer’s weight update must wait until gradients are back-propagated from the output and through all subsequent layers, a phenomenon known as backward-locking; likewise, no weight can update until the full forward/backward pass is completed (update-locking) ( Huo, Gu, and Huang 2018 ; Jaderberg et al. 2017 ) . These locking constraints, together with the neurobiologically implausible requirement of symmetric forward and feedback weights (referred to as the weight transport problem ( Liao, Leibo, and Poggio 2016 ; Shervani-Tabar and Rosenbaum 2023 ) ), introduce significant inefficiencies. Specifically, they slow down training, increase memory consumption, and hinder on-the-fly learning. For instance, in a CLIP-based interactive image-retrieval system ( Wang et al. 2025 ; Zhang et al. 2024 ) , incorporating real-time user feedback would require a full backpropagation pass through hundreds of millions of parameters, an impractical task given BP’s need for global synchronization. Figure 1: Overview of the Eigen Neural Network (ENN) framework. (a) Eigenvector parameterization. Layer \ell decomposes its preactivation weight matrix into an eigenbasis. Z^{(\ell)} collects pre-activation coefficients, \vec{x}^{(\ell-1)} is the input from last layer. Q^{(\ell)} and P^{(\ell)} are orthonormal basis, \Lambda^{(\ell)} is a learned diagonal vector. (b) In a conventional BP pipeline, global classification loss \mathcal{L}_{\text{CLS}} and orthonormality loss \mathcal{L}_{\text{ortho}} are minimized via chained gradients. The ellipsis (…) on each arrow signals that the activation continues through all subsequent layers to the final output. \hat{y} is the ground truth, \vec{y}_{O} is the global output. (c) In ENN’s BP-free mode, each layer predicts locally with logits \vec{y}^{(\ell)} and receives a local error signal \mathcal{L}_{\ell} , enabling weight updates that are fully parallel and require no weight transport or global synchronization. To address these fundamental limitations of BP, we introduced Eigen Neural Networks (ENN) – a novel neural architecture and training paradigm designed for more effective and parallelizable learning. ENN reformulates network weights in a shared eigenbasis: instead of learning each weight parameter directly, the model learns a set of orthonormal eigenbasis vectors and locally trainable coefficients that linearly combine those basis vectors to form the weights of each layer. In other words, each layer’s weight matrix is expressed as a weighted sum of common eigenbasis components. This reparameterization directly targets BP’s inefficiencies. First, by learning in the eigenbasis, ENN ensures that all trainable coefficients receive meaningful gradient signals – no component of the weight space is left with vanishing or neglected updates. The orthonormal eigenbasis provides well-conditioned directions for error propagation, so gradient information is distributed across all coefficients without dying out, eliminating gradient starvation. Second, ENN’s weight update mechanism breaks the backward/update locking of standard BP. Since each layer’s weight coefficients are updated in a decoupled eigenbasis, they can be optimized simultaneously and independently. Layers no longer need to wait for sequential backpropagation; each can apply gradient updates in parallel, free of the global locking that slows ordinary BP. Hence, the ENN architecture naturally integrates into BP-free, block-wise frameworks—referred to as ENN- \ell —significantly enhancing their parallelism and performance. The primary contributions of this research can be summarized as follows: • We introduced Eigen Neural Networks (ENN), a general-purpose deep learning architecture that represents synaptic weights in a shared eigenvector basis, fundamentally altering how weights are parameterized and coupled within each layer. • We embedded the ENN model on top of recent BP-based neural network architectures, significantly improving their performance on both image classification (CIFAR-10, CIFAR-100, Tiny ImageNet, and ImageNet) and cross-modal image-text retrieval tasks (COCO and Flickr30K). • Remarkably, our ENN- \ell surpasses leading BP-free baselines and achieves comparable accuracy to BP training when run on the same backbones. Moreover, ENN- \ell attains over 2\times speedup compared to BP with the same backbone, demonstrating superior parallelism and computational efficiency. Related Work Deep vision models such as ResNet ( He et al. 2016 ) and Vision Transformer (ViT) ( Dosovitskiy et al. 2020 ) owe their success to error backpropagation, but this comes with low training efficiency and biological implausibility. Consequently, two main lines of research have emerged to address BP’s limitations: those that improve its representational quality through regularization, and those that replace it entirely with BP-free learning rules. Regularization methods aim to impose a beneficial structure within the BP framework. For instance, DropCov ( Wang et al. 2022 ) uses adaptive dropout to decorrelate features, while orthogonality-enforcing methods like OCNN ( Wang et al. 2020 ) and SRIP ( Bansal, Chen, and Wang 2018 ) regularize weight matrices to improve feature diversity. Pushing this further, DeCEF ( Yu, Scheidegger, and McKelvey 2024 ) factorizes convolutions into shared eigen-filters to impose a low-rank structure. While these techniques improve feature quality without increasing parameter counts, they remain fundamentally constrained by BP’s procedural bottlenecks, leaving the global, sequential gradient flow untouched.