Differentiating Generalized Eigenvalues and Eigenvectors

Jan de Leeuw · 2025

Read Differentiating Generalized Eigenvalues and Eigenvectors by Jan de Leeuw (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-09355.txt.

Introduction

Jan de Leeuw works out how generalized eigenvalues and related vectors change when the matrices behind them change. These values are central in many data methods, but their rates of change are not always easy to get. He treats cases where matrices depend, in a straight or curved way, on a set of parameters.

The paper gives formulas for first and second rates of change, covering both square and rectangular matrices. It also provides R functions to compute them in general and in special cases. Checks against numeric estimates confirm the formulas. Brief uses in multivariate data analysis show why the derivatives matter in practice.

Catalog record

CREATORJan de Leeuw
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-09355

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Differentiating Generalized Eigenvalues and Eigenvectors Jan de Leeuw Abstract We give formulae for first and second derivatives of generalized eigenvalues/eigenvectors of symmetric matrices and generalized singular values/singular vectors of rectangular matrices when the matrices are linear or nonlinear functions of a vector of parameters. In addition we provide functions in R to compute these derivatives, both in the general case and in various special cases. Formulae are checked against Jacobians and Hessians computed by numerical differentiation. Some applications to multivariate data analysis are discussed. \KOMAoption captionstableheading Note: This is a working manuscript which will be expanded/updated frequently. All suggestions for improvement are welcome. All Rmd, tex, html, pdf, R, and C files are in the public domain. Attribution will be appreciated, but is not required. The files can be found at https://github.com/deleeuw/generalized-eigen-derivatives . 1 Introduction The generalized eigenvalue decomposition (GEVD) problem for a pair (A,B) of square symmetric matrices of order n is to find the solution (X,\Lambda) of the system \displaystyle AX \displaystyle=BX\Lambda, (1a) \displaystyle X^{\prime}BX \displaystyle=I. (1b) We assume that B is positive definite. The matrix X of eigenvectors and the diagonal matrix \Lambda of eigenvalues are both square of order n . As an aside we mention that the generalized eigenvalues of ( 1a ) and ( 1b ) are also the eigenvalues of the asymmetric matrix B^{-1}A and of the symmetric matrix \smash{B^{-1/2}AB^{-1/2}} . GEVD is at the basis of a great many computations in multivariate statistics. In multinormal small sample theory the derivatives are needed for the Jacobians in change-of-variable calculations to find the distribution of many interesting statistics (mathai). The applications we have in mind, however, are in large-sample statistics, where the derivatives are used in Delta Method computations of bias estimates, standard errors, and confidence intervals. The equations ( 1a ) and ( 1b ) implicitly define (X(),\Lambda()) as a function of (A,B) . It turns out that under suitable assumptions on the matrix arguments these implicit functions are actually differentiable, and this makes it interesting to compute their derivatives. There is a humongous and scattered literature on formulae for and computations of derivatives of eigenvalues and eigenvectors in linear algebra, numerical mathematics, engineering, multivariate statistics, and even in physics. Reviewing and collecting all relevant literature is an impossible task. We only give the basic mathematical references, which provide the foundations upon which our results will be built (Kato ( [ ref-kato_84 ] ), Baumgärter ( [ ref-baumgartel_85 ] ), Brustad ( [ ref-brustad_19 ] )). We have written code in R (R Core Team ( [ ref-r_core_team_25 ] )) to implement our final formulae for the first and second derivatives. Code is available at https://github.com/deleeuw/generalized-eigen-derivatives . In addition we have also written R code, available in the same repository, to numerically verify the final formulae. This uses the functions jacobian() and hessian() from the excellent R package numDeriv (Gilbert and Varadhan ( [ ref-gilbert_varadhan_19 ] )). Of course numerical verification, especially in the nonlinear case, is limited to a small number of examples, but if we get the same results as from our formulae we do gain some confidence. I should perhaps also mention a previous version of this paper (De Leeuw ( [ ref-deleeuw_R_07c ] )), which has some errors and is not as complete as the current version. 2 Basic Results Suppose A() and B() are differentiable symmetric matrix valued functions of order n on an open subset \Theta of \mathbb{R}^{p} , and suppose at \theta_{0}\in\Theta the matrix B(\theta_{0}) is positive definite and the generalized eigenvalues of (A(\theta_{0}),B(\theta_{0})) are all different. Then the ordered eigenvalues \Lambda() and the eigenvectors X() are analytic functions of the parameters in a neighborhood of \theta_{0} . We use subscripts i,j=1,\cdots,n for the rows and columns of A and B , subscripts \eta,\nu=1,\cdots,n for the eigenvalues and eigenvectors, and s,t=1,\cdots,p for the parameters in \theta . Thus x_{i\nu} is element i of eigenvector x_{\nu} . The partial derivative of a function A() on \Theta with respect to \theta_{s} , evaluated at \theta_{0} , is written as \mathcal{D}_{s}A(\theta_{0}) . In order not to clutter our formulae the parameter vector where the derivative is evaluated is usually not explicitly specified. Note also that it is sometimes necessary to use parentheses to distinguish (\mathcal{D}_{s}A)x from \mathcal{D}_{s}(Ax) . 2.1 First Partials Differentiate ( 1a ) with respect to \theta_{s} . Then (\mathcal{D}_{s}A)X+A(\mathcal{D}_{s}X)=BX(\mathcal{D}_{s}\Lambda)+B(\mathcal{D}_{s}X)\Lambda+(\mathcal{D}_{s}B)X\Lambda. (2) Premultiplying ( 2 ) by X^{\prime} and rearranging gives \mathcal{D}_{s}\Lambda=\{X^{\prime}(\mathcal{D}_{s}A)X-X^{\prime}(\mathcal{D}_{s}B)X\Lambda\}+\{\Lambda X^{\prime}B(\mathcal{D}_{s}X)-X^{\prime}B(\mathcal{D}_{s}X)\Lambda\}. (3) The matrix \Lambda X^{\prime}B(\mathcal{D}_{s}X)-X^{\prime}B(\mathcal{D}_{s}X)\Lambda is anti-symmetric and consequently has a zero diagonal. Taking the diagonal 1 1 1 The diagonal \text{diag}(X) of a square matrix X is a diagonal matrix with the same diagonal as X . on both sides of ( 3 ) gives \mathcal{D}_{s}\Lambda=\text{diag}\{X^{\prime}(\mathcal{D}_{s}A)X-X^{\prime}(\mathcal{D}_{s}B)X\Lambda\}, (4) or, for a single eigenvalue \lambda_{\nu} with corresponding eigenvector x_{\nu} , \mathcal{D}_{s}\lambda_{\nu}=x_{\nu}^{\prime}(\mathcal{D}_{s}A-\lambda_{\nu}\mathcal{D}_{s}B)x_{\nu}. (5) Taking the off-diagonal 2 2 2 The off-diagonal \text{off}(X) of a square matrix X is X with its diagonal replaced by zeroes. on both sides of ( 3 ) gives \text{off}\{\Lambda X^{\prime}B(\mathcal{D}_{s}X)-X^{\prime}B(\mathcal{D}_{s}X)\Lambda\}=-\text{off}\{X^{\prime}(\mathcal{D}_{s}A)X-X^{\prime}(\mathcal{D}_{s}B)X\Lambda\}. (6) X is non-singular, and thus there is a unique square H_{s} such that \mathcal{D}_{s}X=XH_{s} . Using this substitution ( 6 ) becomes \text{off}\{\Lambda H_{s}-H_{s}\Lambda\}=-\text{off}\{X^{\prime}(\mathcal{D}_{s}A)X-X^{\prime}(\mathcal{D}_{s}B)X\Lambda\}. (7) Switch to subscript notation and solve for H_{s} . For \nu\neq\eta \{H_{s}\}_{\nu\eta}=-\frac{x_{\nu}^{\prime}\{\mathcal{D}_{s}A-\lambda_{s}\mathcal{D}_{s}B\}x_{\eta}}{\lambda_{\nu}-\lambda_{\eta}}. (8) This does not give a value for the diagonal of H_{s} . Differentiating ( 1b ) gives X^{\prime}B(\mathcal{D}_{s}X)+(\mathcal{D}_{s}X)^{\prime}BX+X^{\prime}\mathcal{D}_{s}BX=0. (9) Using \mathcal{D}_{s}X=XH_{s} and taking the diagonal of ( 9 ) gives h_{\nu\nu}=-\tfrac{1}{2}x_{\nu}^{\prime}(\mathcal{D}_{s}B)x_{\nu}. (10) Combining ( 8 ) and ( 10 ) shows that for the eigenvector corresponding with \lambda_{\nu} we have \mathcal{D}_{s}x_{\nu}=-\sum_{\begin{subarray}{c}\eta=1\\ \eta\neq\nu\end{subarray}}^{n}\frac{x_{\eta}^{\prime}\{\mathcal{D}_{s}A-\lambda_{\nu}\mathcal{D}_{s}B\}x_{\nu}}{\lambda_{\eta}-\lambda_{\nu}}x_{\eta}-\tfrac{1}{2}x_{\nu}^{\prime}(\mathcal{D}_{s}B)x_{\nu}\cdot x_{\nu}, (11) which can also be written as \mathcal{D}_{s}x_{\nu}=-\sum_{\begin{subarray}{c}\eta=1\\ \eta\neq\nu\end{subarray}}^{n}\left\{\frac{x_{\eta}x_{\eta}^{\prime}}{\lambda_{\eta}-\lambda_{\nu}}\right\}(\mathcal{D}_{s}A-\lambda_{\nu}\mathcal{D}_{s}B)x_{\nu}-\tfrac{1}{2}(x_{\nu}^{\prime}(\mathcal{D}_{s}B)x_{\nu})x_{\nu}. (12) It is notationally convenient to have a matrix expression for this derivative. Define the matrices (A-\lambda_{\nu}B)^{-}:=\sum_{\begin{subarray}{c}\eta=1\\ \eta\neq\nu\end{subarray}}^{n}\left\{\frac{x_{\eta}x_{\eta}^{\prime}}{\lambda_{\eta}-\lambda_{\nu}}\right\}=X(\Lambda-\lambda_{\nu}I)^{+}X^{\prime}, (13) where (\Lambda-\lambda_{\nu}I)^{+} is the Moore-Penrose inverse of \Lambda-\lambda_{\nu}I . Matrix ( 13 ) is a reflexive g-inverse of A-\lambda_{\nu}B (Rao and Mitra ( [ ref-rao_mitra_71 ] ), section 2.5). Of the four Penrose conditions only the first two are satisfied. To verify this we use definition ( 13 ) and A-\lambda_{\nu}B=X^{-T}(\Lambda-\lambda_{\nu}I)X^{-1} . We find \displaystyle(A-\lambda_{\nu}B)(A-\lambda_{\nu}B)^{-}(A-\lambda_{\nu}B) \displaystyle=A-\lambda_{\nu}B, (14a) \displaystyle(A-\lambda_{\nu}B)^{-}(A-\lambda_{\nu}B)(A-\lambda_{\nu}B)^{-} \displaystyle=(A-\lambda_{\nu}B)^{-}, (14b) \displaystyle(A-\lambda_{\nu}B)(A-\lambda_{\nu}B)^{-} \displaystyle=X^{-T}(I-e_{\nu}e_{\nu}^{\prime})X^{\prime}, (14c) \displaystyle(A-\lambda_{\nu}B)^{-}(A-\lambda_{\nu}B) \displaystyle=X(I-e_{\nu}e_{\nu}^{\prime})X^{-1}. (14d) We see that (A-\lambda_{\nu}B)^{-} is a Moore-Penrose inverse of A-\lambda_{\nu}B if and only if B is identically equal to one (and thus X^{\prime}X=XX^{\prime}=I ). Using ( 13 ) gives \mathcal{D}_{s}x_{\nu}=-(A-\lambda_{\nu}B)^{-}(\mathcal{D}_{s}A-\lambda_{\nu}\mathcal{D}_{s}B)x_{\nu}-\tfrac{1}{2}(x_{\nu}^{\prime}(\mathcal{D}_{s}B)x_{\nu})x_{\nu}. (15) The equations ( 5 ) and ( 15 ) will be used frequently throughout this paper. For eigenvalues ( 5 ) shows that computing \mathcal{D}_{s}\lambda_{\nu} only requires us to know \lambda_{\nu} and x_{\nu} , not the other eigenvalues and eigenvectors. Equation ( 15 ) suggests the same thing is true for \mathcal{D}_{s}x_{\nu} , but this is only apparent, because we need all generalized eigenvalues and eigenvectors to compute (A-\lambda_{\nu}B)^{-} . If we only need the derivatives of, say, the first few eigenvectors from a very large system this can be computationally quite expensive. There is an alternative way of computing \mathcal{D}_{s}x_{\nu} which does not require a full GEVD. We know that \mathcal{D}_{s}x_{\nu} is a solution of the linear equations (A-\lambda_{\nu}B)\mathcal{D}_{s}x_{\nu}=-(\mathcal{D}_{s}A-\lambda_{\nu}\mathcal{D}_{s}B)x_{\nu}+(\mathcal{D}_{s}\lambda_{\nu})Bx_{v}. (16) Because \lambda_{\nu} is a simple eigenvalue, the matrix A-\lambda_{\nu}B is of rank n-1 , with a null space consisting of all vectors proportional to x_{\nu} . Write y_{\nu} for the right hand side of

Read the complete text