The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning
Youqi Wu, Farzan Farnia · 2026
Read The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning by Youqi Wu, Farzan Farnia (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-02117.txt.
Introduction
Youqi Wu and colleagues extend the minimax maximum entropy principle to von Neumann entropy. That gives a game-theoretic case for VNE maximization. Von Neumann entropy is a core quantum quantity, lately used as a spectral gauge of diversity. While VNE maximization under constraints is well known, no principled classical analogue has been built out for data-driven contexts. Their view yields a robust reading of maximum VNE solutions. They test the principle on two problems: picking a kernel representation from embeddings, and completing matrices from partially observed entries.
Catalog record
| CREATOR | Youqi Wu, Farzan Farnia |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52854 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602.02117
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The Maximum von Neumann Entropy Principle: Theory and Applications in Machine Learning Youqi Wu, Farzan Farnia Abstract Von Neumann entropy (VNE) is a fundamental quantity in quantum information theory and has recently been adopted in machine learning as a spectral measure of diversity for kernel matrices and kernel covariance operators. While maximizing VNE under constraints is well known in quantum settings, a principled analogue of the classical maximum entropy framework, particularly its decision theoretic and game theoretic interpretation, has not been explicitly developed for VNE in data driven contexts. In this paper, we extend the minimax formulation of the maximum entropy principle due to Grünwald and Dawid to the setting of von Neumann entropy, providing a game-theoretic justification for VNE maximization over density matrices and trace-normalized positive semidefinite operators. This perspective yields a robust interpretation of maximum VNE solutions under partial information and clarifies their role as least committed inferences in spectral domains. We then illustrate how the resulting Maximum VNE principle applies to modern machine learning problems by considering two representative applications, selecting a kernel representation from multiple normalized embeddings via kernel-based VNE maximization, and completing kernel matrices from partially observed entries. These examples demonstrate how the proposed framework offers a unifying information-theoretic foundation for VNE-based methods in kernel learning. 1 Introduction Entropy is a fundamental quantity in information theory, quantifying the uncertainty of a random variable and playing a central role in data compression, communication, and statistical inference. In the classical setting, Shannon entropy [ 34 ] provides a mathematical characterization of uncertainty for a probability distribution. In quantum information theory [ 36 , 24 ] , this notion is extended to quantum states, which are represented by density matrices. A density matrix \rho is a positive semidefinite (PSD) operator with unit trace, and its associated entropy, known as the von Neumann entropy (VNE) , is defined as the Shannon entropy of its eigenvalues: S(\rho)\;=\;-\mathrm{Tr}(\rho\log\rho)\;=\;-\sum_{i}\lambda_{i}(\rho)\log\lambda_{i}(\rho), (1) where the eigenvalues \{\lambda_{i}(\rho)\} form a probability distribution. Recently, VNE has been adopted in machine learning as a tool for measuring diversity and effective dimensionality of data representations [ 2 , 10 ] . In particular, kernel matrices and kernel covariance operators, when normalized to have unit trace, naturally define density-matrix-like objects whose spectra encode global similarity structure [ 16 , 27 ] . This observation has led to the use of VNE-based quantities, such as the Vendi score [ 10 ] , as spectral diversity measures for datasets and generative models. However, in several machine learning applications, the underlying density matrix is not fully observed, for example due to missing entries of the target kernel matrix, noisy measurements, or kernel covariance operators estimated from limited data. This motivates a fundamental question: how should VNE-based analysis be carried out when the density matrix is only partially specified? In classical information theory, entropy-based statistical inference under partial knowledge can be performed by the maximum entropy principle . Introduced by Jaynes in [ 18 ] , this principle advocates selecting, among all probability distributions consistent with the given constraints, the distribution with the maximum Shannon entropy value. A canonical example is that the Gaussian distribution maximizes (differential) entropy among all distributions with fixed mean and covariance matrix [ 6 ] . Beyond its variational formulation, the maximum entropy principle admits an operational interpretation as demonstrated by Grünwald and Dawid in [ 12 ] , showing that the maximum entropy solution arises as a minimax-optimal strategy in a game-theoretic setting where a decision maker competes against an adversary over an ambiguity set of distributions. In this paper, we extend this framework to von Neumann entropy, with a focus on machine learning settings involving partial knowledge of a density matrix. Given an ambiguity set \mathcal{C} of density matrices consistent with the available information, we consider the maximum VNE principle , i.e. to choose and base the inference on the density matrix \rho^{*} with the largest VNE: \rho^{\star}\;\in\;\arg\max_{\rho\in\mathcal{C}}S(\rho). (2) Our main contribution is to formally show that the game-theoretic interpretation of the maximum entropy principle developed by Grünwald and Dawid in [ 12 ] extends naturally to the matrix-valued setting. This yields a principled minimax justification for maximizing VNE over the set of density matrices, directly paralleling the classical case for probability distributions. We further show that this minimax framework applies beyond VNE to other spectral entropy functionals of interest. In particular, we consider matrix-based Rényi entropies, including the quadratic Rényi entropy of order two and the Rényi entropy of order infinity, which is governed by the maximum eigenvalue [ 31 , 33 ] . We also extend the framework to conditional settings, drawing on minimax formulations of supervised learning [ 9 ] , thereby enabling applications in which kernel objects arise jointly from inputs and outputs. Finally, we illustrate the proposed maximum VNE principle through two concrete applications in machine learning. The first concerns the selection of a kernel representation from multiple normalized embeddings. By considering convex combinations of the corresponding kernel matrices and applying the Max-VNE principle over the mixture weights, we obtain a principled method for selecting a least-committal kernel mixture that promotes spectral diversity. The second application addresses kernel matrix completion from partial observations, where we select, among all feasible positive semidefinite completions consistent with the observed entries, the one that maximizes VNE. We complement the theoretical developments with numerical experiments on standard image datasets, demonstrating the practical relevance of the maximum VNE principle in data-driven settings. 2 Related Work The maximum entropy principle originates from Shannon’s entropy [ 34 ] and Jaynes’ proposal of selecting the least-committal model under a group of constraints [ 18 ] . In more recent literature, closely related principles replace entropy maximization by minimizing dependence subject to partial constraints. Notable examples include the minimum mutual information principle, which generalizes maximum entropy to discriminative learning by minimizing I(X;Y) under marginal and feature constraints [ 11 ] , and the minimum maximal correlation principle, which selects joint distributions minimizing the Hirschfeld–Gebelein–Rényi (HGR) correlation under low-order constraints [ 8 , 30 ] . These dependence-minimization principles admit minimax interpretations and lead to robust learning rules when only partial information is available. Complementarily, the minimax and game-theoretic foundation of maximum entropy itself is formalized by Grünwald and Dawid [ 12 ] . These works motivate extending entropy- and dependence-based minimax principles from probability distributions to matrix-valued entropy objectives. In quantum information theory, uncertainty is described by density matrices, which are positive semidefinite, unit-trace operators, and von Neumann entropy extends Shannon entropy to this operator-valued setting via the eigenvalue distribution [ 36 , 24 ] . Maximum entropy reasoning for quantum systems is well established, particularly in quantum statistical mechanics and inference, where it is used to select a density matrix consistent with partial information. These works motivate studying maximum entropy principles directly at the level of density matrices, but they are typically grounded in physical constraints rather than data-driven ambiguity sets. Von Neumann entropy and related spectral entropy functionals have recently gained prominence in machine learning as measures of diversity and effective dimensionality for kernel matrices and kernel covariance operators. The Vendi score defines diversity as the exponential of the von Neumann entropy of a normalized similarity matrix and has been