Revisiting Orbital Minimization Method for Neural Operator Decomposition
J. Jon Ryu , Samuel Zhou , Gregory W. Wornell Department of EECS, MIT, Cambridge, MA 02139, United States { jongha , samtzhou , gww }@mit.edu · 2025
Read Revisiting Orbital Minimization Method for Neural Operator Decomposition by J. Jon Ryu , Samuel Zhou , Gregory W. Wornell Department of EECS, MIT, Cambridge, MA 02139, United States { jongha , samtzhou , gww }@mit.edu (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-21952.txt.
Introduction
J. Jon Ryu, Samuel Zhou, and Gregory W. Wornell revive an old method from computational chemistry. They put it to work training neural networks. The method dates from the nineteen nineties. It was built for eigenvalue problems, under the name orbital minimization.
Splitting a linear operator into its main modes underlies much of machine learning. It supports representation learning, dynamics, and equation solving. Neural approximations promise to make that splitting scale. Ryu and his co-authors prove the old objective is consistent. Their proof uses plain linear algebra. They also trace links to ideas that surfaced separately in several fields. Then they adapt the method for positive semidefinite operators. Benchmark tasks show practical advantages. The moral is archival: classic numerical methods can earn a second life inside modern learning pipelines.
Catalog record
| CREATOR | J. Jon Ryu , Samuel Zhou , Gregory W. Wornell Department of EECS, MIT, Cambridge, MA 02139, United States { jongha , samtzhou , gww }@mit.edu |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 86 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-21952
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Orbital Minimization Method for Neural Operator Decomposition J. Jon Ryu , Samuel Zhou , Gregory W. Wornell Department of EECS, MIT, Cambridge, MA 02139, United States { jongha , samtzhou , gww }@mit.edu Abstract Spectral decomposition of linear operators plays a central role in many areas of machine learning and scientific computing. Recent work has explored training neural networks to approximate eigenfunctions of such operators, enabling scalable approaches to representation learning, dynamical systems, and partial differential equations (PDEs). In this paper, we revisit a classical optimization framework from the computational physics literature known as the orbital minimization method (OMM), originally proposed in the 1990s for solving eigenvalue problems in computational chemistry. We provide a simple linear-algebraic proof of the consistency of the OMM objective, and reveal connections between this method and several ideas that have appeared independently across different domains. Our primary goal is to justify its broader applicability in modern learning pipelines. We adapt this framework to train neural networks to decompose positive semidefinite operators, and demonstrate its practical advantages across a range of benchmark tasks. Our results highlight how revisiting classical numerical methods through the lens of modern theory and computation can provide not only a principled approach for deploying neural networks in numerical simulation, but also effective and scalable tools for machine learning. 1 Introduction Spectral decomposition of linear operators is a foundational tool in applied mathematics, with far-reaching implications across machine learning (ML) and scientific computing. Eigenfunctions of differential, integral, and graph-based operators capture essential geometric and dynamical structures, playing a central role in problems ranging from representation learning to solving partial differential equations (PDEs). As such, the design of efficient and scalable methods for approximating these eigenfunctions has become a core pursuit in modern computational science. Recent advances have leveraged neural networks to approximate spectral components of operators, offering promising avenues for scaling classical techniques to high-dimensional or irregular domains in scientific simulation. Notable examples include recent breakthroughs in quantum chemistry based on neural-network ansatzes [ 41 , 9 , 42 ] . Beyond quantum chemistry, neural approaches have also proven effective in diverse domains such as spectral embeddings [ 15 , 43 ] , Koopman operator theory for analyzing dynamical systems [ 32 , 23 , 20 ] , and neural solvers for PDEs [ 26 ] . For an overview of the broader literature, we refer the reader to [ 43 , Appendix B] . However, many existing methods rely on surrogate losses or architectural constraints that lack a clear variational foundation, often leading to brittle optimization or limited extensibility. In this work, we revisit a classical optimization framework from computational quantum chemistry known as the orbital minimization method (OMM). Originally developed in the 1990s for finding the ground state in electronic-structure theory [ 38 , 34 , 33 , 37 , 1 ] , OMM offers a direct and elegant approach to approximating the top eigenspace of a positive-definite operator, without requiring explicit orthonormalization. Despite its heuristic, domain-specific origins, we derive OMM rigorously from a simple variational principle valid for general positive-semidefinite matrices and operators. We leverage this reinterpretation to construct a modern neural variant of OMM, suitable for decomposing a wide class of linear operators. Our contributions are threefold: 1. We provide a new derivation of the OMM objective with a simple linear-algebraic proof for finite-dimensional cases, clarifying its key idea and extending its theoretical foundation. When specialized for streaming PCA, we identify its connection to the celebrated Sanger’s rule, which is often known as the generalized Hebbian algorithm [ 44 ] . 2. We adapt this framework to train neural networks that learn eigenspaces of positive-definite operators, eliminating the need for explicit orthogonalization or eigensolvers. 3. We empirically demonstrate the effectiveness of our method across a range of tasks, including learning Laplacian-based representations in reinforcement learning settings, solving PDEs, and self-supervised-learning representation of images and graphs. We emphasize that properly positioning the OMM within the modern ML literature is of both theoretical and practical significance. Although the resulting formulation may appear natural in hindsight, recent work in ML has proposed a variety of alternative, and often considerably more complex, approaches for computing eigenvectors of matrices or eigenfunctions of operators. In this work, we draw connections between OMM and several existing lines of research, including streaming PCA [ 44 ] from theoretical statistics and computational science, as well as recent neural parameterizations used for operator learning in deep learning [ 43 ] , and independent development of spectral techniques in reinforcement learning literature [ 51 , 13 ] . These connections provide a unified perspective on seemingly disparate methods. Our findings underscore the value of revisiting classical numerical techniques through the lens of modern ML, offering both conceptual clarity and practical advantages for scalable spectral learning. Notation. For an integer n\geq 1 , [n]\mathrel{\mathop{\mathchar 58\relax}}=\{1,\ldots,n\} . We use capital sans-serif fonts, such as \mathsf{A} and \mathsf{V} , to denote matrices. In particular, \mathsf{I}_{k}\in\mathbb{R}^{k\times k} denotes the identity matrix. For a square matrix \mathsf{A} , \mathsf{tr}(\mathsf{A}) denotes its trace. For a matrix \mathsf{V}\in\mathbb{R}^{d_{1}\times d_{2}}=[{\bf v}_{1},\ldots,{\bf v}_{d_{2}}] , where {\bf v}_{j}\in\mathbb{R}^{d_{1}} for each j\in[d_{2}] , we use the subscript notation \mathsf{V}_{1\mathchar 58\relax k}\mathrel{\mathop{\mathchar 58\relax}}=[{\bf v}_{1},\ldots,{\bf v}_{k}] for 1\leq k\leq d_{2} to denote the subset of columns. 2 Methods In this section, we revisit the orbital minimization method and provide a new variational perspective for the finite-dimensional matrix case. We then present its application to an infinite-dimensional problem, i.e., an operator problem. 2.1 Preliminary: Matrix Eigenvalue Problem Let \mathsf{A}\in\mathbb{R}^{d\times d} be a real symmetric matrix of interest. 1 1 1 We focus on real matrices for simplicity, but the technique can be naturally adapted for Hermitian matrices. We wish to compute the top- k eigenvectors of \mathsf{A} . By the spectral theorem, \mathsf{A} admits an eigenvalue decomposition, i.e., there exists an orthogonal eigenbasis \mathsf{W}=[{\bf w}_{1},\ldots,{\bf w}_{d}]\in\mathbb{R}^{d\times d} such that we can write \mathsf{A}=\mathsf{W}\Lambda\mathsf{W}^{\intercal}=\sum_{i=1}^{d}\lambda_{i}{\bf w}_{i}{\bf w}_{i}^{\intercal} , where \Lambda\mathrel{\mathop{\mathchar 58\relax}}=\mathsf{diag}(\lambda_{1},\ldots,\lambda_{d}) , where \lambda_{1}\geq\lambda_{2}\geq\ldots\geq\lambda_{d} . Our goal is to find the top- k eigensubspace, i.e., \spn\{{\bf w}_{1},\ldots,{\bf w}_{k}\} , and ideally the top- k eigenvectors {\bf w}_{1},\ldots,{\bf w}_{k} and corresponding eigenvalues \lambda_{1},\ldots,\lambda_{k} . We use a shorthand notation a\wedge b\mathrel{\mathop{\mathchar 58\relax}}=\min\{a,b\} for a,b\in\mathbb{R} . For a symmetric matrix \mathsf{A}\in\mathbb{R}^{d\times d} , the most standard variational characterization of the top- k eigensubspace is the multicolumn Rayleigh quotient maximization formulation, which is \max_{\mathsf{V}\in\mathbb{R}^{d\times k}\mathchoice{\colon}{\colon}{\mathchar 58\relax\mspace{1mu}}{\mathchar 58\relax}\mathsf{V}^{\intercal}\mathsf{V}=\mathsf{I}_{k}}\mathsf{tr}(\mathsf{V}^{\intercal}\mathsf{A}\mathsf{V}). Note that it is a natural extension of the maximum Rayleigh quotient characterization of the largest eigenvalue. While we can derive its unconstrained version as \max_{\mathsf{V}\in\mathbb{R}^{d\times k}}\mathsf{tr}((\mathsf{V}^{\intercal}\mathsf{V})^{-1}\mathsf{V}^{\intercal}\mathsf{A}\mathsf{V}), the inversion (\mathsf{V}^{\intercal}\mathsf{V})^{-1} renders the optimization less practical. For example, if \mathsf{V} is not full rank (or nearly degenerate), computing \left(\mathsf{V}^{\intercal}\mathsf{V}\right)^{-1} may be infeasible or numerically unstable. 2.2 Orbital Minimization Method: Matrix Version We now restrict our focus on a positive-semidefinite (PSD) matrix \mathsf{A}\in\mathbb{R}^{d\times d} . The orbital minimization method (OMM) [ 38 , 37 , 34 , 33 , 30 ] aims to solve the following maximization problem: \min_{\mathsf{V}\in\mathbb{R}^{d\times k}}\mathcal{L}_{\mathsf{omm}}(\mathsf{V}),\quad\text{where }\mathcal{L}_{\mathsf{omm}}(\mathsf{V})\mathrel{\mathop{\mathchar 58\relax}}=-\mathsf{tr}((2\mathsf{I}_{k}-\mathsf{V}^{\intercal}\mathsf{V})\mathsf{V}^{\intercal}\mathsf{A}\mathsf{V}) While it is an unconstrained objective, it is widely known that its minimizer \mathsf{V}^{\star} satisfies \mathsf{V}^{\star}(\mathsf{V}^{\star})^{\intercal}=\mathsf{W}_{1\mathchar 58\relax k}\mathsf{W}_{1\mathchar 58\relax k}^{\intercal} , i.e., \mathsf{V} corresponds to the top- k eigenvectors up to rotation. Interestingly, the OMM objective does not have any spurious