Information Geometry and Asymptotic Theory for SMML Estimators

Enes Makalic, Daniel Francis Schmidt · 2026

Read Information Geometry and Asymptotic Theory for SMML Estimators by Enes Makalic, Daniel Francis Schmidt (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-05241.txt.

Introduction

Makalic and Schmidt show that the goal splits into claim entropy and cross-entropy, weighing naming cost against coding cost. For a fixed split, each cell's point is the spread cutting gap from that cell's data spread. Under shape, optimal SMML splits approach pullbacks, through the fit gauge, of weighted Voronoi tilings, with claim chances as weights. For exp groups, SMML points satisfy a moment rule and act as KL/Bregman centroids, while cells are pullbacks of convex shapes in data space.

Catalog record

CREATOREnes Makalic, Daniel Francis Schmidt
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE48369
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.05241

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Information Geometry and Asymptotic Theory for SMML Estimators Enes Makalic, Daniel Francis Schmidt Abstract We develop an asymptotic theory for strict minimum message length (SMML) estimators in regular parametric models with countable data spaces. We show that, asymptotically, the optimal SMML partition is induced by a weighted Fisher–Voronoi tessellation in parameter space, pulled back through the maximum likelihood estimator. We further show that each SMML codepoint is asymptotically a weighted average of the maximum likelihood estimates associated with observations in its cell. These results imply that the SMML estimator is consistent and converges at the usual parametric n − 1 / 2 n^{-1/2} rate under standard regularity conditions. We also give a Kullback–Leibler projection interpretation of SMML codepoints and a decomposition of the expected SMML codelength into an assertion entropy and an expected conditional cross-entropy. In exponential families, the theory simplifies further: SMML codepoints satisfy a moment-matching condition, and optimal SMML cells are induced by a polyhedral partition of sufficient-statistic space. K eywords strict minimum message length \cdot asymptotic theory \cdot information geometry \cdot Fisher–Rao distance \cdot Kullback–Leibler projection \cdot exponential family 1 Introduction Let \{X_{i}\}_{i=1}^{n} be independent and identically distributed random variables from a parametric family \{p(\mathbf{x}|\bm{\theta}):\bm{\theta}\in\Theta\subseteq\mathbb{R}^{p}\} . We observe a data set \mathbf{x}=(x_{1},\ldots,x_{n})\in\mathcal{X}_{n} , where the sample space \mathcal{X}_{n} is assumed to be countable, possibly infinite. Let \mathcal{P}=\{P_{1},\ldots,P_{k}\} denote a partition of \mathcal{X}_{n} into k non-empty, pairwise disjoint cells whose union is \mathcal{X}_{n} . For each cell P_{j} , define \displaystyle q_{j}=\sum_{\mathbf{y}\in P_{j}}r(\mathbf{y}),\qquad r(\mathbf{x})=\int_{\Theta}p_{n}(\mathbf{x}|\bm{\theta})\pi(\bm{\theta})\,d\bm{\theta},\qquad j=1,\ldots,k, (1) where \pi(\bm{\theta}) is a prior density on \Theta , r(\mathbf{x}) is the resulting marginal distribution of the data, and q_{j} is the assertion probability of the cell P_{j} and its associated codepoint \bm{\theta}_{j}^{*} . Throughout, we restrict attention to partitions whose cells have positive marginal mass, so that q_{j}>0 for all j , and to partitions with finitely many cells, so that the associated codebook is finite and the expected two-part codelength is well defined as a finite sum. We will use p_{n}(\mathbf{x}|\bm{\theta}) to denote the joint likelihood of the full data set \mathbf{x}=(x_{1},\ldots,x_{n}) . Under strict minimum message length (SMML) encoding [ 11 , 14 , 13 , 15 ] , the expected two-part codelength associated with a partition \mathcal{P} is \displaystyle\mathcal{I}(\mathcal{P})=\underbrace{-\sum_{j=1}^{k}q_{j}\log q_{j}}_{\text{assertion}}-\underbrace{\sum_{j=1}^{k}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\log p_{n}(\mathbf{x}|\bm{\theta}_{j}^{*})}_{\text{detail}}. (2) Unless otherwise stated, all logarithms are natural, so codelengths are measured in nats (or nits). For each cell P_{j} , the associated SMML codepoint \bm{\theta}_{j}^{*} is defined to minimise the expected detail codelength within that cell, equivalently, \displaystyle\bm{\theta}_{j}^{*}=\argmin_{\bm{\theta}\in\Theta}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}=\argmax_{\bm{\theta}\in\Theta}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\log p_{n}(\mathbf{x}|\bm{\theta}). (3) The SMML problem is to determine the partition \mathcal{P}^{*} that minimises ( 2 ) over a prescribed class \Pi of partitions of \mathcal{X}_{n} : \displaystyle\mathcal{P}^{*}=\argmin_{\mathcal{P}\in\Pi}\mathcal{I}(\mathcal{P}). (4) The resulting partition, together with its associated codepoints \Theta^{*}=\{\bm{\theta}_{1}^{*},\ldots,\bm{\theta}_{k}^{*}\} , defines the SMML estimator [ 12 , 13 , 15 ] . When referring to the optimal SMML partition \mathcal{P}^{*} , we will write q_{j}^{*} (or q_{j,n}^{*} in the sample-size-dependent setting) to denote the optimal assertion probabilities. The aim of this paper is to develop an asymptotic theory for strict minimum message length (SMML) estimators in regular parametric models. We make four main contributions. First, we give an information-geometric characterisation of optimal SMML partitions, showing that, asymptotically, they are induced by weighted Fisher–Voronoi tessellations in parameter space, pulled back through the maximum likelihood estimator. Second, we show that each SMML codepoint admits an asymptotic weighted-average representation in terms of the maximum likelihood estimates associated with observations in its cell. Third, we use these results to establish consistency of the SMML estimator and its convergence at the usual parametric rate. Fourth, for exponential family models, we show that the SMML partition is induced by a polyhedral partition of sufficient-statistic space. The remainder of the paper is organised as follows. Section 3 introduces the regularity conditions and the asymptotic quantisation framework used throughout. Section 4 establishes the connection between SMML partitions and weighted Fisher–Voronoi tessellations, and derives the weighted-average representation of SMML codepoints. Section 5 proves consistency and the associated rate of convergence of the SMML estimator. Section 6 specialises the general theory to exponential families and describes the resulting polyhedral structure in sufficient-statistic space. Section 7 concludes with a brief discussion of implications and possible extensions. 2 Interpretations of SMML This section discusses two useful reformulations of the SMML criterion. The first shows that, for a fixed partition cell, the associated SMML codepoint is a Kullback–Leibler projection [ 7 ] of the normalised cellwise distribution onto the model family. The second rewrites the expected SMML codelength as the sum of an assertion entropy and an expected conditional cross-entropy [ 9 ] , thereby making explicit the two-stage structure of SMML encoding. 2.1 SMML codepoints as Kullback–Leibler projections Fix a partition \mathcal{P}=\{P_{1},\ldots,P_{k}\} of \mathcal{X}_{n} , and consider a single cell P_{j} . Recall that the corresponding SMML codepoint is defined by \displaystyle\bm{\theta}_{j}^{*}=\argmin_{\bm{\theta}\in\Theta}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}. (5) Define the normalised cellwise distribution \displaystyle\bar{p}_{j}(\mathbf{x})=\frac{r(\mathbf{x})}{q_{j}}\,\mathds{1}_{\{\mathbf{x}\in P_{j}\}},\qquad q_{j}=\sum_{\mathbf{y}\in P_{j}}r(\mathbf{y})>0. (6) Then ( 5 ) may be rewritten as \displaystyle\bm{\theta}_{j}^{*}=\argmin_{\bm{\theta}\in\Theta}\sum_{\mathbf{x}\in\mathcal{X}_{n}}\bar{p}_{j}(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}, (7) where \mathds{1}_{\{{\bf x}\in P_{j}\}} is the indicator function equal to 1 if {\bf x}\in P_{j} , and zero otherwise. Using the identity \displaystyle\sum_{\mathbf{x}\in\mathcal{X}_{n}}\bar{p}_{j}(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}=H(\bar{p}_{j})+D_{\mathrm{KL}}(\bar{p}_{j}\|p_{\bm{\theta}}), where H(\bar{p}_{j}) is the Shannon entropy of \bar{p}_{j} [ 9 ] and D_{\mathrm{KL}} denotes Kullback–Leibler divergence [ 7 ] , we obtain the following characterisation. Proposition 1 (SMML codepoints as KL projections) For a fixed partition cell P_{j} , the associated SMML codepoint satisfies \displaystyle\bm{\theta}_{j}^{*}=\argmin_{\bm{\theta}\in\Theta}D_{\mathrm{KL}}(\bar{p}_{j}\|p_{\bm{\theta}}). (8) That is, \bm{\theta}_{j}^{*} is the Kullback–Leibler projection of the normalised cellwise distribution \bar{p}_{j} onto the model family \{p(\cdot|\bm{\theta}):\bm{\theta}\in\Theta\} . Proof. By ( 5 ) and ( 6 ), \displaystyle\bm{\theta}_{j}^{*} \displaystyle=\argmin_{\bm{\theta}\in\Theta}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}=\argmin_{\bm{\theta}\in\Theta}q_{j}\sum_{\mathbf{x}\in\mathcal{X}_{n}}\bar{p}_{j}(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}. Since q_{j}>0 does not depend on \bm{\theta} , the minimiser is unchanged if it is removed. The result then follows from the decomposition \displaystyle\sum_{\mathbf{x}\in\mathcal{X}_{n}}\bar{p}_{j}(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta})\bigr\}=H(\bar{p}_{j})+D_{\mathrm{KL}}(\bar{p}_{j}\|p_{\bm{\theta}}), where H(\bar{p}_{j}) is constant in \bm{\theta} . \square Proposition 1 provides a useful interpretation of the SMML codepoint: within each cell, SMML chooses the parameter value whose model distribution is closest, in Kullback–Leibler divergence, to the normalised cellwise distribution. In this sense, the SMML codepoint is a cellwise information projection of the normalised cellwise distribution onto the parametric family. 2.2 A decomposition of the SMML codelength We next rewrite the expected SMML codelength in a form that makes explicit its two-part coding structure. Let J be a discrete random variable taking values in \{1,\ldots,k\} with \displaystyle\mathbb{P}(J=j)=q_{j}>0,\qquad j=1,\ldots,k, (9) where q_{j} is the assertion probability of cell P_{j} . The assertion part of the SMML codelength is the Shannon entropy of the cell probabilities: \displaystyle-\sum_{j=1}^{k}q_{j}\log q_{j}=H(J)=H(q), (10) where H(q) denotes the entropy of the discrete distribution (q_{1},\ldots,q_{k}) ; in particular, H(q)\leq\log k. Thus the expected codelength of the assertion grows at most logarithmically in the number of codepoints. For the detail part of the codelength, using ( 6 ) we obtain \displaystyle-\sum_{j=1}^{k}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\log p_{n}(\mathbf{x}|\bm{\theta}_{j}^{*}) \displaystyle=\sum_{j=1}^{k}q_{j}\left\{H(\bar{p}_{j})+D_{\mathrm{KL}}(\bar{p}_{j}\|p_{\bm{\theta}_{j}^{*}})\right\}. (11) Equivalently, if we write \displaystyle H(\bar{p}_{j},p_{\bm{\theta}_{j}^{*}}):=\sum_{\mathbf{x}\in\mathcal{X}_{n}}\bar{p}_{j}(\mathbf{x})\bigl\{-\log p_{n}(\mathbf{x}|\bm{\theta}_{j}^{*})\bigr\}, (12) for the cross-entropy of \bar{p}_{j} relative to p(\cdot|\bm{\theta}_{j}^{*}) , then \displaystyle-\sum_{j=1}^{k}\sum_{\mathbf{x}\in P_{j}}r(\mathbf{x})\log p_{n}(\mathbf{x}|\bm{\theta}_{j}^{*})=\mathbb{E}\!\left[H(\bar{p}_{J},p_{\bm{\theta}_{J}^{*}})\right]. (13) Combining ( 10 ) and ( 13 ) yields the decomposition \displaystyle\mathcal{I}(\mathcal{P})=H(q)+\mathbb{E}\!\left[H(\bar{p}_{J},p_{\bm{\theta}_{J}^{*}})\right]. (14) Equation ( 14 ) makes explicit the hierarchical, two-stage structure of SMML encoding. In the first stage, the index J of the cell is encoded using a code of expected length H(q) . In

Read the complete text