Exponential Family Discriminant Analysis: Generalizing LDA-Style Generative Classification to Non-Gaussian Models
Anish Lakkapragada · 2026
Read Exponential Family Discriminant Analysis: Generalizing LDA-Style Generative Classification to Non-Gaussian Models by Anish Lakkapragada (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-20655.txt.
Introduction
Lakkapragada introduces EFDA, a wider form of linear discriminant analysis for classification. Classic LDA assumes Gaussian data for each class. EFDA lets each class follow any shared exponential family law. Its decision rule stays linear in the sufficient statistic. LDA returns as a special case, while curved boundaries can appear in raw features. Tests on five data laws matched rival accuracy. Calibration error fell two to six fold. All four main claims were also machine checked in Lean 4.
Catalog record
| CREATOR | Anish Lakkapragada |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47814 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.20655
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Exponential Family Discriminant Analysis: Generalizing LDA-Style Generative Classification to Non-Gaussian Models Thanks: Code available at https://github.com/anish-lakkapragada/EFDA . Anish Lakkapragada Abstract We introduce Exponential Family Discriminant Analysis (EFDA), a unified generative framework that extends classical Linear Discriminant Analysis (LDA) beyond the Gaussian setting to any member of the exponential family. Under the assumption that each class-conditional density belongs to a common exponential family, EFDA derives closed-form maximum-likelihood estimators for all natural parameters and yields a decision rule that is linear in the sufficient statistic, recovering LDA as a special case and capturing nonlinear decision boundaries in the original feature space. We prove that EFDA is asymptotically calibrated and statistically efficient under correct specification, and we generalise it to K ≥ 2 K\geq 2 classes and multivariate data. Through extensive simulation across four exponential-family distributions (Weibull, Gamma, Exponential, Poisson), EFDA matches the classification accuracy of LDA, QDA, and logistic regression while reducing Expected Calibration Error (ECE) by 2 2 – 6 × 6\times , a gap that is structural : it persists for all n n and across all class-imbalance levels, because misspecified models remain asymptotically miscalibrated. We further prove and empirically confirm that EFDA’s log-odds estimator approaches the Cramér–Rao bound under correct specification, and is the only estimator in our comparison whose mean squared error converges to zero. Complete derivations are provided for nine distributions. Finally, we formally verify all four theoretical propositions in Lean 4, using Aristotle (Harmonic) and OpenGauss (Math, Inc.) as proof generators, with all outputs independently machine-checked by AXLE (Axiom). 1 Introduction Generative classification proceeds by modelling class-conditional densities f_{0}(x)=p(X\mid Y=0) and f_{1}(x)=p(X\mid Y=1) , then applying Bayes’ rule to obtain the posterior p(Y=1\mid X) . Linear Discriminant Analysis (LDA) [ 5 ] is the canonical generative classifier: it places Gaussian densities on each class (with shared covariance), solves for maximum-likelihood (MLE) parameters in closed form, and produces a log-odds function that is linear in the feature vector. LDA’s Gaussian assumption is both its strength and its principal limitation. When the true class-conditional distributions are non-Gaussian, LDA’s probability estimates are miscalibrated, and the linear log-odds boundary may fail to separate the classes. The discriminative alternative (logistic regression) avoids the Gaussian assumption but constrains the log-odds to be linear in the feature vector X , not in some transformation of it. Consequently, both methods fail to capture the true decision boundary whenever the log-odds is a nonlinear function of X . Exponential families encompass an extremely broad class of distributions. They share the canonical density form f(\mathbf{x}\mid\boldsymbol{\eta})=h(\mathbf{x})\exp\!\bigl(\boldsymbol{\eta}\cdot T(\mathbf{x})-A(\boldsymbol{\eta})\bigr), (1) where T(\mathbf{x}) is the sufficient statistic , \boldsymbol{\eta} the natural parameter , A(\boldsymbol{\eta}) the log-partition function, and h(\mathbf{x}) a base measure. Members include the Normal, Gamma, Poisson, Weibull, Bernoulli, Negative Binomial, and many others. Exponential families arise naturally in probabilistic modelling. Kernel-based methods [ 7 , 11 ] apply the exponential family to dimensionality reduction. Robust variants such as GLD [ 4 ] and FEMDA [ 6 ] extend discriminant analysis to contaminated or heterogeneous Gaussian/elliptical data. Contributions. 1. EFDA (Sections 3 – 3.2 ). We derive a generative classifier for binary and K -class settings in which each class-conditional distribution belongs to the same exponential family. EFDA fits natural parameters by closed-form (or single-equation) MLE and produces a decision rule linear in T(\mathbf{x}) . 2. Theory (Section 4 ). We concretely theorize under regularity assumptions that EFDA is (i) calibrated under correct specification, (ii) Bayes-optimal asymptotically, and (iii) achieves the Cramér–Rao information bound for natural-parameter estimation. 3. Derivations (Section 6 ). Closed-form EFDA for nine distributions: Normal (two forms), Laplace, Exponential, Gamma, Weibull, Poisson, Bernoulli, Negative Binomial (Table 1 ). 4. Experiments (Section 7 ). Comprehensive evaluation across four distributions: EFDA matches accuracy of all baselines while reducing ECE by 2 – 5\times . We ablate class imbalance, sample size, and unknown shape parameters; and demonstrate multi-class EFDA on K\in\{3,5\} . 5. Statistical Efficiency (Section 8 ). A formal asymptotic efficiency analysis proves and empirically validates that EFDA’s log-odds estimator attains the Cramér–Rao bound and is the only classifier evaluated here whose MSE converges to zero. 6. Formal Verification (Section 9 ). All four theoretical propositions are formally verified in Lean 4. We compare two AI-assisted proof generators (Aristotle by Harmonic and OpenGauss by Math, Inc.), with all outputs independently machine-checked by AXLE (Axiom). 2 Background 2.1 Linear Discriminant Analysis Let X\in\mathbb{R}^{p} be the feature vector and Y\in\{0,\ldots,K-1\} the class label. We aim to model the posterior P(Y=k\mid X) . LDA assumes X\mid Y=k\;\sim\;\mathcal{N}(\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}),\quad k=0,\ldots,K-1, with a shared covariance \boldsymbol{\Sigma} . Given dataset \mathcal{D}=\{(X_{i},Y_{i})\}_{i=1}^{n} , the MLE yields \hat{\alpha}_{k}=\frac{N_{k}}{n},\quad\hat{\boldsymbol{\mu}}_{k}=\frac{1}{N_{k}}\!\sum_{i:Y_{i}=k}\!X_{i},\quad\hat{\boldsymbol{\Sigma}}=\frac{1}{n}\sum_{k}\sum_{i:Y_{i}=k}(X_{i}-\hat{\boldsymbol{\mu}}_{k})(X_{i}-\hat{\boldsymbol{\mu}}_{k})^{\!\top}. The binary log-odds ratio is linear in X : \log\frac{P[Y=1\mid X]}{P[Y=0\mid X]}=\underbrace{\log\frac{\alpha_{1}}{\alpha_{0}}-\tfrac{1}{2}(\boldsymbol{\mu}_{1}+\boldsymbol{\mu}_{0})^{\top}\boldsymbol{\Sigma}^{-1}(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{0})}_{\beta_{0}}+\underbrace{(\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{0})^{\top}\boldsymbol{\Sigma}^{-1}}_{\boldsymbol{\beta}^{\top}}X. Logistic regression posits this same linear form and fits \beta_{0},\boldsymbol{\beta} directly, without any distributional assumption. 2.2 Exponential Family Distributions A distribution with density ( 1 ) belongs to the exponential family. Key properties used throughout: • Moment identity. \mathbb{E}_{\eta}[T(\mathbf{x})]=\nabla_{\eta}A(\eta) and \operatorname{Cov}_{\eta}[T(\mathbf{x})]=\nabla^{2}_{\eta}A(\eta) . Used in Section 3 to derive the EFDA MLE condition ( 3 ) and in Section 4 to establish asymptotic calibration (Proposition 2 ). • Sufficient statistic. T(\mathbf{x}) contains all information about \boldsymbol{\eta} ; the conditional \mathbf{x}\mid T(\mathbf{x}) does not depend on \boldsymbol{\eta} . Used in Sections 3 – 3.2 to characterise the EFDA decision boundary as linear in T(\mathbf{x}) , and in Section 4 for the efficiency result (Proposition 3 ). • Convexity. A(\boldsymbol{\eta}) is convex, so the log-likelihood is concave in \boldsymbol{\eta} . Used in Section 3 to guarantee a unique MLE solution and in Section 4 (Proposition 1 ) for consistency. These properties are standard; see, e.g., 8 for a detailed treatment. 3 Exponential Family Discriminant Analysis 3.1 Binary EFDA We assume that for each class k\in\{0,1\} , the class-conditional density is f_{k}(\mathbf{x}\mid\boldsymbol{\eta}_{k})=h(\mathbf{x})\exp\!\bigl(\boldsymbol{\eta}_{k}\cdot T(\mathbf{x})-A(\boldsymbol{\eta}_{k})\bigr), with the same h , T , and A across classes but different natural parameters \boldsymbol{\eta}_{0}\neq\boldsymbol{\eta}_{1} . Given \mathcal{D}=\{(X_{i},Y_{i})\}_{i=1}^{n} , the complete-data log-likelihood is \displaystyle\mathcal{L}(\alpha,\boldsymbol{\eta}_{0},\boldsymbol{\eta}_{1}) \displaystyle=\sum_{i=1}^{n}\log h(X_{i})+\mathbf{1}[Y_{i}=1]\bigl[\log\alpha+\boldsymbol{\eta}_{1}\cdot T(X_{i})-A(\boldsymbol{\eta}_{1})\bigr] \displaystyle\quad+\mathbf{1}[Y_{i}=0]\bigl[\log(1-\alpha)+\boldsymbol{\eta}_{0}\cdot T(X_{i})-A(\boldsymbol{\eta}_{0})\bigr]. (2) Setting partial derivatives to zero yields the MLE conditions (full derivation in Appendix A ): \boxed{\hat{\alpha}=\frac{N_{1}}{n},\qquad\frac{1}{N_{k}}\sum_{i:\,Y_{i}=k}\!T(X_{i})=\nabla_{\!\boldsymbol{\eta}_{k}}A(\boldsymbol{\eta}_{k}),\quad k\in\{0,1\}.} (3) The MLE condition for \boldsymbol{\eta}_{k} says: the sample mean of T(X_{i}) in class k equals the model’s expected sufficient statistic at \boldsymbol{\eta}_{k} . This is the method-of-moments identity for exponential families. Because A is distribution-specific, the closed-form solution for \hat{\boldsymbol{\eta}}_{k} varies by family; Table 1 collects the solutions for eight common distributions. Log-odds formula. Once parameters are estimated, the log-odds ratio admits the clean form (derived in Appendix B ): \ell(\mathbf{x})=\log\frac{P[Y=1\mid X]}{P[Y=0\mid X]}=\underbrace{\log\frac{\alpha}{1-\alpha}+A(\boldsymbol{\eta}_{0})-A(\boldsymbol{\eta}_{1})}_{\text{intercept}}+\underbrace{(\boldsymbol{\eta}_{1}-\boldsymbol{\eta}_{0})}_{\text{``slope''}}\cdot T(\mathbf{x}). (4) This is linear in the sufficient statistic T(\mathbf{x}) . When T(\mathbf{x})=\mathbf{x}/\sigma (Normal, known \sigma ), this reduces to LDA’s linear boundary. For non-Gaussian distributions (e.g. Weibull where T(x)=x^{k} , or Poisson where T(x)=x but the boundary is non-trivial in x ), EFDA captures decision regions that are nonlinear in the original feature space. 3.2 Multi-class EFDA The binary derivation extends naturally. For K classes with priors \alpha_{k} and natural parameters \boldsymbol{\eta}_{k} , the MLE conditions become \hat{\alpha}_{k}=\frac{N_{k}}{n},\qquad\frac{1}{N_{k}}\sum_{i:\,Y_{i}=k}\!T(X_{i})=\nabla_{\!\boldsymbol{\eta}_{k}}A(\boldsymbol{\eta}_{k}),\quad k=0,\ldots,K-1. (5) Classification uses the MAP rule: \hat{Y}(\mathbf{x})=\operatorname*{arg\,max}_{k}\Bigl[\log\hat{\alpha}_{k}+\boldsymbol{\eta}_{k}\cdot T(\mathbf{x})-A(\boldsymbol{\eta}_{k})\Bigr]. (6) The pairwise log-odds between classes j and k is again linear in T(\mathbf{x}) : \log\frac{P[Y=j\mid\mathbf{x}]}{P[Y=k\mid\mathbf{x}]}=\log\frac{\alpha_{j}}{\alpha_{k}}+[A(\boldsymbol{\eta}_{k})-A(\boldsymbol{\eta}_{j})]+(\boldsymbol{\eta}_{j}-\boldsymbol{\eta}_{k})\cdot T(\mathbf{x}). This constitutes a generalised linear classifier in sufficient-statistic space . The connection to Naive Bayes follows directly from the MAP rule