Optimal information deletion and Bayes’ theorem

Hans Montcho, Håvard Rue · 2026

Read Optimal information deletion and Bayes’ theorem by Hans Montcho, Håvard Rue (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09061.txt.

Introduction

What spread should replace a Bayesian posterior when some data must be forgotten? Hans Montcho and two co-authors revisit Zellner's information-processing view of Bayes' rule, in which it processes prior and data information without loss. For ordinary Bayesian inference, they show the information-dropping goal is met by the leave-data-out spread, the answer retraining would give.

The work extends to generalized Bayesian updating based on loss functions, where coherent sequential dropping yields exactly the spread from retained information alone. An axiomatic theory for possibilistic dropping yields a principled rule for exact unlearning of that kind. Restricting the exact rules to tractable families yields variational and maxitive methods as natural approximations.

Catalog record

CREATORHans Montcho, Håvard Rue
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09061

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Optimal information deletion and Bayes’ theorem Hans Montcho, Håvard Rue Abstract In this same journal, Arnold Zellner published a seminal paper on Bayes’ theorem as an optimal information processing rule. This result led to the variational formulation of Bayes’ theorem, which is the central idea in generalized variational inference. Almost 40 years later, we revisit these ideas, but from the perspective of information deletion. We investigate rules which update a posterior distribution into an antedata distribution when a portion of data is removed. In such context, a rule which does not destroy or create information is called the optimal information deletion rule and we prove that it coincides with the traditional use of Bayes’ theorem. Keywords: Information theory, Bayesian unlearning. 1 Introduction In 1988 , Arnold Zellner published a paper entitled Optimal information processing and Bayes’ theorem . In that seminal paper ( Zellner 1988 ) , he proved from first principles that Bayes’ theorem, more than the simple application of conditional probability rules, is the optimal way to update a prior information into a posterior information in light of new data. This result led to the variational formulation of Bayes’ theorem, which has recently seen a resurgence and is the central idea in generalized variational inference ( Bissiri et al. 2016 , Knoblauch et al. 2022 ) . Almost 40 years later, we revisit these ideas, but from the perspective of data removal. Although the original motivation for this work is cross validation in statistics or data deletion in computer science, a general framework of information deletion is used to investigate a rule which updates a posterior distribution into an antedata distribution when a subset of data is removed. In such context, a rule which does not destroy or create information is called the optimal information deletion rule and coincides with the traditional use of Bayes’ theorem. In this work, we prove this new result with the same variational calculus’ formalism used by Zellner. The structure of this work is fully inspired by Zellner’s paper; and any similarity is not just a coincidence, but intentionally by design. Therefore, we organize it as follows: in section 2 , we briefly review the ideas in Zellner 1988 , then in section 3 we define the data removal concept, formalize it as an information deletion problem, and derive the functional to minimize in order to conserve information. In section 4 , we obtain the minimum of the previous functional, and in section 5 , we prove that this minimum coincides with the solution obtained via Bayes’ theorem. Finally, in section 6 we discuss some implications of these results. 2 Optimal information processing rule Zellner started by hypothesizing a parametric statistical model \pi(y|\theta) indexed by \theta\in\Theta , for given data y , he then assumed a distribution \pi(\theta|I) for \theta based on prior information I , and together the prior and likelihood represent the input information that needs to be processed. The outputs are given by what he called the postdata distribution q(\theta|y) , and the probability distribution for the data \displaystyle\pi(y|I)=\int\pi(y|\theta)\pi(\theta|I)d\theta . An information processing rule (IPR) is a procedure that transforms input into output information, and the optimal IPR is one that does not destroy or add some nonexistent information. We depict the IPR in figure 1 . Inputs: \pi(\theta|I) \pi(y|\theta) Information Processing Rule Outputs: q(\theta|y) \pi(y|I) Figure 1 : Illustration of the Information Processing Rule An important remark is the difference in terminology, as Zellner did not name q(\theta|y) , the posterior distribution. In fact, one of the most important results from the paper is that the most efficient IPR, i.e. the best q^{*}(\theta|y) is exactly the posterior distribution obtained via Bayes’ theorem q^{*}(\theta|y)=\pi(\theta|y)=\pi(\theta|I)\pi(y|\theta)\bigg/\int_{\Theta}\pi(\theta|I)\pi(y|\theta) , making Bayes’ theorem 100\% IPR efficient. For this purpose, he used the Shannon’s entropy to measure the information content, then defined the information loss as \Delta[q(\theta|y)]=(\text{Output Information}-\text{Input Information}) in Equation ( 1 ), and finally derived the optimal IPR by minimizing \Delta[q(\theta|y)] over the space of all possible distributions q(\theta|y) . \displaystyle\Delta[q(\theta|y)] \displaystyle= Output Information - Input Information \displaystyle= \displaystyle\mathbb{E}_{q(\theta|y)}\bigg[\log q(\theta|y)+\log\pi(y|I)\bigg] \displaystyle- \displaystyle\mathbb{E}_{q(\theta|y)}\bigg[\log\pi(\theta|I)+\log\pi(y|\theta)\bigg] \displaystyle\Delta[q(\theta|y)] \displaystyle= \displaystyle\mathbb{E}_{q(\theta|y)}\bigg[\log\frac{q(\theta|y)}{\pi(\theta|I)}\frac{\pi(y|I)}{\pi(y|\theta)}\bigg]. (1) Using a variational calculus formalism, Zellner proved that the optimal IPR in Equation ( 2 ) coincides exactly with Bayes’ theorem. \displaystyle q^{*}(\theta|y)=\pi(\theta|I)\pi(y|\theta)\bigg/\int_{\Theta}\pi(\theta|I)\pi(y|\theta)d\theta. (2) An interesting implication of this result comes from the interpretation of Bayes’ theorem as the solution of an infinite dimensional optimization problem, therefore approximate solutions could be obtained by restricting the optimization space. More specifically, optimizing over a smaller space \mathcal{Q} gives an approximate \tilde{q}(\theta|y) in Equation ( 2 ) and represents a formal justification for variational inference ( Blei et al. 2017 ) as an approximate information processing rule. \displaystyle\tilde{q}(\theta|y) \displaystyle= \displaystyle\underset{q\;\in\;\mathcal{Q}}{\text{argmin}}\;\Delta[q(\theta|y)] \displaystyle= \displaystyle\underset{q\;\in\;\mathcal{Q}}{\text{argmin}}\;\mathbb{E}_{q}\bigg[\log\frac{q(\theta|y)}{\pi(\theta|I)}\frac{\pi(y|I)}{\pi(y|\theta)}\bigg] \displaystyle= \displaystyle\underset{q\;\in\;\mathcal{Q}}{\text{argmin}}\;-\mathbb{E}_{q}\left[\log(y|\theta)\right]+\mathbb{E}_{q}\bigg[\log\frac{q(\theta|y)}{\pi(\theta|y)}\bigg]+\mathbb{E}_{q}[\pi(y|I)] \displaystyle\tilde{q}(\theta|y) \displaystyle= \displaystyle\underset{q\;\in\;\mathcal{Q}}{\text{argmin}}\;-\mathbb{E}_{q}\left[\log(y|\theta)\right]+\text{KL}\left\{q|\pi(\theta|I)\right\}+C. (3) In the rest of this work we answer the backward question: what is the optimal information processing rule if one seeks to remove some information from the posterior distribution given the complete data ? we call such a rule the optimal information deletion rule 3 Information concept and optimal information deletion rule We motivate the need for data removal with two problems, namely, cross validation and data deletion. In cross validation ( Vehtari & Ojanen 2012 ) , a subset of the data is used to fit a statistical model, while the left out subset is used to assess the quality of the fitted model using a scoring rule, and the procedure is repeated over a designed partition of the complete data. To avoid multiple model refits, we usually resort to approximate cross validation and aim for an approximate leave-data-out posterior distribution. In comparison, in data deletion ( Bourtoule et al. 2021 ) , the goal is to delete user information, not only from databases, but also from a statistical model already fitted and deployed, without refitting it from scratch. Both problems share a commonality, the need to update a posterior distribution given the complete data after removing some information, i.e. some data. The strategy to obtain the optimal information deletion rule (IDR) is similar to the IPR in Zellner 1988 and requires the definition of input and output information. Then, the optimal IDR is defined as the one that minimizes some information loss. The inputs to our IDR are the outputs from the IPR in section 2 , that is, the posterior distribution given the complete data \pi(\theta|y) and the distribution for the data \pi(y|I) given prior information I . We divide the data into two parts y=(y_{g},y_{-g}) , where y_{g} represents the subset whose information content needs to be removed from \pi(\theta|y) , and we denote its likelihood contribution by \pi(y_{g}|\theta) , which is also an input. Using \pi(y_{g}|\theta) as the likelihood contribution for y_{g} implicitly assumes conditional independence, but this is purely for readability and is not required at all, as we explain in Appendix 7.4 . Regarding the outputs, after removing the information content of y_{g} , we are left with the distribution for the rest of the data \displaystyle\pi(y_{-g}|I)=\int_{\Theta}\pi(y_{-g}|\theta)\pi(\theta|I)d\theta given prior information I , and the antedata distribution q(\theta|y_{-g}) . We summarize the components of the information deletion process in figure 2 . Inputs: \pi(\theta|y) \pi(y|I) \pi(y_{g}|\theta) Information Deletion Rule Outputs: q(\theta|y_{-g}) \pi(y_{-g}|I) Figure 2 : Illustration of

Read the complete text