Riemannian Optimization for Hadamard Products of Low-Rank Matrices

Pratik Jawanpuria, Ankish Chandresh, Bamdev Mishra · 2026

Read Riemannian Optimization for Hadamard Products of Low-Rank Matrices by Pratik Jawanpuria, Ankish Chandresh, Bamdev Mishra (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01216.txt.

Introduction

Jawanpuria, Chandresh, and Mishra study a compact matrix model. The model takes the elementwise product of two low-rank matrices. It fits data with multiplicative structure well. But linked row and column scalings add extra symmetry. That symmetry makes the model hard to fit. The team frames the fit as a search on a curved space. That space is a Riemannian quotient manifold.

They build a new metric from the Frobenius inner product. The metric stays fixed under the linked scalings. Their gradient method uses a Gauss-Newton step size. It needs no tuning and scales in a linear way. Tests on real and fake data show the approach works well.

Catalog record

CREATORPratik Jawanpuria, Ankish Chandresh, Bamdev Mishra
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01216

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Riemannian Optimization for Hadamard Products of Low-Rank Matrices Pratik Jawanpuria, Ankish Chandresh, Bamdev Mishra Abstract The elementwise Hadamard product of two low-rank matrices provides a parameter-efficient model for data with multiplicative structure, but its modeling is challenging due to the presence of additional symmetries under coupled row/column scalings between the two factors. In order to leverage the geometry of the space, we formulate the learning of such matrices as optimization on a Riemannian quotient manifold. We propose a novel block-diagonal Riemannian metric derived from the pullback of the Frobenius inner product. The metric is shown to be invariant under the full symmetry group. We develop a Riemannian gradient descent algorithm that uses a tuning-free Gauss–Newton step size and scales linearly in the number of observed entries per iteration. Experiments on real and synthetic datasets illustrate the efficacy of our proposed Riemannian approach. I Introduction Low-rank matrix models are central to modern machine learning, underpinning everything from classical recommender systems [ 11 ] to representation learning [ 13 ] and the parameter-efficient fine-tuning of large-scale models [ 9 ] . However, in many complex domains, the underlying data structure is best captured not by a single low-rank matrix, but by the elementwise (Hadamard) product of two low-rank factors [ 5 ] . For instance, in collaborative filtering, one factor pair can model overall user/item activity levels while the other captures nuanced preference structures [ 11 ] . Similar decoupled interactions arise in link prediction – where community structure is separated from node-level heterogeneity [ 8 ] – and in count data modeling for single-cell genomics [ 17 ] and topic models [ 12 ] . Formally, we consider the target matrix W modeled as: W=(GH^{\top})\circ(UV^{\top}), (1) where G\in\mathbb{R}^{m\times r_{1}} , H\in\mathbb{R}^{n\times r_{1}} , U\in\mathbb{R}^{m\times r_{2}} , V\in\mathbb{R}^{n\times r_{2}} , and \circ denotes the Hadamard (elementwise) product. For brevity, we write X=GH^{\top} and Y=UV^{\top} for the two low-rank factors, so that W=X\circ Y . The Hadamard product of a rank- r_{1} and a rank- r_{2} matrix can have rank as high as r_{1}r_{2} [ 5 ] . A standard factorization W=LR^{\top} achieving this rank would require (m{+}n)(r_{1}r_{2}) parameters, whereas the Hadamard form ( 1 ) requires only (m{+}n)(r_{1}{+}r_{2}) , a reduction when (r_{1}{-}1)(r_{2}{-}1)>1 . However, this structural efficiency introduces multiple optimization challenges. The Hadamard product induces symmetries beyond the standard general linear (GL) reparameterization invariance of fixed-rank factorizations. Specifically, alongside the GL invariance of each factor pair, the product W is invariant under coupled row and column scalings between the two factors. The additional degrees of freedom result in a challenging optimization landscape, especially for standard Euclidean gradient methods. The limits of existing solvers. Ciaperoni et al. [ 5 ] proposed an alternating gradient descent (AGD) method for learning the low-rank Hadamard model ( 1 ). However, AGD is based on Euclidean gradient descent and it ignores the per-row scaling and the highly curved geometry of the Hadamard factor space. Wertz et al. [ 20 ] proposed a block coordinate descent (BCD) method for learning the low-rank Hadamard model with least-squares loss. In this setting, BCD cyclically updates each factor by solving per-row normal equations (first-order optimality conditions) in closed form. Thus, the BCD solver [ 20 ] is quite effective with the least-squares loss but for the non-quadratic losses–which are essential for robust statistics, logistic link prediction, or cross-entropy objectives–BCD’s sub-problems lack closed-form solutions. However, gradient-descent-based approaches such as AGD [ 5 ] can be applied to any smooth loss. Our approach based on Riemannian optimization. In this paper, we bridge this gap by elevating the problem of learning Hadamard product of low-rank matrices to an optimization problem on a Riemannian manifold. To this end, we propose a Riemannian quotient manifold that takes into account the invariances that exist naturally in the Hadamard product of low-rank matrices ( 1 ). Concretely, our contributions are: 1. Geometric characterization: We formally describe the quotient manifold structure of W=(GH^{\top})\circ(UV^{\top}) , identifying both the GL reparameterization and the novel cross-Hadamard scaling symmetries (Section III ). 2. A Novel Riemannian Metric: We propose a block-diagonal Riemannian metric that is scale invariant (Section III ). We also discuss the relationship between our proposed approach and BCD’s normal-equation matrices that are used in its updates [ 20 ] . 3. Riemannian Gradient Descent (RGD): We derive an RGD algorithm equipped with a Gauss-Newton Cauchy step size that requires no hyperparameter tuning, achieving an O(Nr+(m{+}n)r^{4}) per-iteration cost, where N is the number of observed entries (dataset size in learning problems) and r=\max(r_{1},r_{2}) ensuring linear scaling with N . 4. We demonstrate experimentally that RGD performs comparably to BCD on least-squares tasks (across three real datasets) and show that RGD achieves the lowest RMSE on MovieLens-1M across all tested Hadamard rank configurations (Section V ). II Background and related work Block coordinate descent (BCD) [ 20 ] . Given a target matrix A\in\mathbb{R}^{m\times n} and the least squares loss \tfrac{1}{2}\|A-W\|_{F}^{2} where the matrix W is modeled as a Hadamard product of (unknown) low-rank matrices ( 1 ) GH^{\top} and UV^{\top} , each sub-problem in BCD method [ 20 ] reduces to a row-by-row least-squares system. The update equation of the i -th row in the G matrix, denoted as G_{i,:} , is G_{i,:}\leftarrow\arg\min_{g\in\mathbb{R}^{r_{1}}}\tfrac{1}{2}\bigl\|\bigl(A_{i,:}-g(H\operatorname{Diag}(Y_{i,:}))^{\top}\bigr)\bigr\|^{2}, (2) where Y=UV^{\top} as defined in Section I . The normal-equation matrix for this sub-problem is H^{\top}\operatorname{Diag}(Y_{i,:}^{2})H , an r_{1}\times r_{1} matrix that encodes how the current states of H and Y influence the update directions for G . However, this efficiency is strictly bound to the squared-error loss. BCD’s sub-problems lack closed-form solutions for other popular loss functions such as the cross-entropy loss, rendering the approach inapplicable. Conversely, AGD [ 5 ] accommodates arbitrary smooth losses but relies on uniform Euclidean steps, failing to adapt to the highly coupled, factor-dependent geometry captured by normal-equation matrices. Riemannian optimization on quotient manifolds. Riemannian optimization provides a principled framework for unconstrained optimization over smooth manifolds [ 1 , 4 ] . A Riemannian solver requires three core components: a smooth manifold \mathcal{M} , a Riemannian metric g_{x} on the tangent spaces to define gradients, and a retraction to map steps back to \mathcal{M} . For the fixed-rank manifold with factorizations W=LR^{\top} , the quotient geometry under \text{GL}(r) invariance is well studied [ 19 , 15 ] . The choice of the Riemannian metric is critical. While the Euclidean metric ignores factorization symmetries, a full pullback metric captures all curvature at the cost of coupling factor blocks and yielding dense gradient systems. A practical choice is to retain only the block-diagonal elements of the pullback metric [ 14 ] . In particular, Mishra and Sepulchre [ 16 ] showed that tailoring the Riemannian metric to the problem’s underlying structure–such as utilizing normal-equation matrices–acts as a preconditioner for the optimization, significantly accelerating convergence over Euclidean metrics. Generic implementations of these concepts exist in solvers like Manopt [ 3 ] , Pymanopt [ 18 ] , and Manopt.jl [ 2 ] . Our work extends this quotient manifold framework beyond a single low-rank factor to the Hadamard product of two low-rank factors. III Manifold Geometry and Metric We consider optimization problems of the form \min_{G,H,U,V}\;f\!\big(\Phi(G,H,U,V)\big), where \Phi(G,H,U,V)\triangleq(GH^{\top})\circ(UV^{\top})=W and f\colon\mathbb{R}^{m\times n}\to\mathbb{R} is any

Read the complete text