Contrastive Entropy Bounds for Density and Conditional Density Decomposition

Bo Hu, José C. Príncipe · 2025

Read Contrastive Entropy Bounds for Density and Conditional Density Decomposition by Bo Hu, José C. Príncipe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-12903.txt.

Introduction

Bo Hu and Jose C. Principe reinterpret two classic neural network tools through a Gaussian, Bayesian lens. In their view, training a model tightens a probabilistic bound, and the learned density is what makes the bound tight. Mixture density networks use a bound for the marginal distribution, while autoencoders use a conditional one. The paper focuses on networks with multiple outputs, where each output is the center of a Gaussian component, and asks what objective such networks should really optimize.

Their first result equates the autoencoder objective with maximizing the trace of a Gaussian operator. When one-to-one matching is not required, maximizing the operator's nuclear norm instead would raise its overall rank. Their second construction adds a norm term that increases sample diversity and blocks output collapse. The paper is theoretical, offering new objectives for density modeling.

Catalog record

CREATORBo Hu, José C. Príncipe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE180 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-12903

Browse the full library · About the Tower of Babel