The Gaussian-Multinoulli Restricted Boltzmann Machine A Potts Model Extension of the GRBM

Nikhil Kapasi, William Whitehead, Luke Theogarajan · 2025

Read The Gaussian-Multinoulli Restricted Boltzmann Machine A Potts Model Extension of the GRBM by Nikhil Kapasi, William Whitehead, Luke Theogarajan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-11635.txt.

Introduction

Nikhil Kapasi, William Whitehead, Luke Theogarajan, and a co-author introduce a new kind of restricted Boltzmann machine for data with clear, grouped states. Older models use hidden units that are simply on or off. That simple form can struggle with tasks that need richer, multi-valued ideas.

Their model replaces those binary units with units that can take several states, drawn from the Potts model. The paper derives the energy, the learning rules, and training tricks that stop units from collapsing to one state. On memory and recall tests, the new model matched or beat the older one at the same size and cost.

Catalog record

CREATORNikhil Kapasi, William Whitehead, Luke Theogarajan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-11635

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Gaussian-Multinoulli Restricted Boltzmann Machine A Potts Model Extension of the GRBM Nikhil Kapasi, William Whitehead, Luke Theogarajan Abstract Many real-world tasks, from associative memory to symbolic reasoning, demand discrete, structured representations that standard continuous latent models struggle to express naturally. We introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM), a generative energy-based model that extends the Gaussian-Bernoulli RBM (GB-RBM) by replacing binary hidden units with q q -state Potts variables. This modification enables a combinatorially richer latent space and supports learning over multivalued, interpretable latent concepts. We formally derive GM-RBM’s energy function, learning dynamics, and conditional distributions, showing that it preserves tractable inference and training through contrastive divergence. Empirically, we demonstrate that GM-RBMs model complex multimodal distributions more effectively than binary RBMs, outperforming them on tasks involving analogical recall and structured memory. Our results highlight GM-RBMs as a scalable framework for discrete latent inference with enhanced expressiveness and interoperability. 1 Introduction Restricted Boltzmann Machines (RBMs) constitute a class of undirected energy-based models defined on a bipartite graph between visible units \mathbf{v} and latent units \mathbf{h} (hidden). Their tractable block Gibbs updates [ 6 ] and scalable approximate inference via contrastive divergence enable efficient training on high-dimensional datasets [ 1 , 6 ] . In the binary hidden configuration, an RBM reduces to an Ising model where hidden units h_{j}\in{-1,+1} interact pairwise through couplings W_{ij} , yielding an energy landscape analogous to classical spin systems [ 14 ] . The Gaussian–Bernoulli RBM (GB-RBM) augments this formulation by replacing these visible variables with Gaussian potentials, while preserving binary hidden spins [ 7 ] . Although GB-RBMs capture continuous inputs such as image intensities or sensor readings, the binary latent constraints limit their expressivity when representing multimodal or discrete feature correlations [ 4 , 15 ] . The q -state Potts model generalizes the Ising model by allowing each spin s_{j} to occupy one of q discrete states, enriching the model’s capacity to encode complex interaction patterns and phase transitions [ 22 , 32 ] . Potts-type energy functions have demonstrated efficacy in image segmentation, combinatorial optimization, community detection, and categorical embedding tasks, where capturing higher-order discrete structures is key [ 12 ] . Building on these principles, we introduce the Gaussian–Multinoulli Restricted Boltzmann Machine (GM-RBM), which integrates Gaussian-distributed visible units with a hidden layer of q -state Potts spins. This GB-RBM-like architecture provides a clear path to benchmarking and showcasing the power of Potts-based energy functions. The Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM) we propose has a small change in the underlying representation of hidden nodes that has a profound improvement on model performance. While prior work has addressed mixed-variate inputs using Gaussian-Bernoulli structures [ 26 ] , convolutional and tiled weight-sharing schemes for spatial hierarchies [ 10 ] , and implicit mixture models of RBMs for multimodal density estimation [ 20 ] , these approaches retain binary or sigmoidal-based latent representations, limiting their ability to capture categorical variation at the feature level. In contrast, the GM-RBM replaces binary hidden units with q -state Potts (Multinoulli) variables, enabling each hidden unit to express one of several discrete latent modes rather than mere presence or absence. This yields a latent space that scales as q^{m} (with m hidden units), and facilitates more structured, interpretable, and compositional generative behavior. Unlike energy-based models that treat discrete latents as auxiliary sampling artifacts [ 4 ] , or continuous latent embeddings for mixed data [ 25 ] , the GM-RBM directly leverages categorical latent dynamics to build rich distributions over continuous-valued visible units. This hybrid formulation opens new avenues for modeling structured analog data, such as trajectories, natural images, or biophysical time series, with discrete control over generative features. 2 Background and Model Overview 2.1 Background and Motivation Restricted Boltzmann Machines (RBMs) have played a foundational role in energy-based modeling and unsupervised representation learning. In particular, Gaussian-Bernoulli RBMs (GB-RBMs), which pair real-valued Gaussian visible units with binary latent variables, have been adopted for modeling continuous input data such as images and signals [ 15 ] . However, while GB-RBMs are capable of learning low-level features, they face inherent limitations in their ability to capture structured, discrete latent representations. The root of this limitation lies in the mismatch between the binary latent space and the Gaussian visible manifold. The Gaussian prior enforces a continuous energy landscape in the input space, which, when combined with binary latent units, leads to the learning of clustered, overlapping encodings. These encodings tend to coalesce into fuzzy representations that lack the expressiveness needed for capturing discrete abstractions, such as object parts, symbolic states, or multimodal feature variants. Moreover, the binary latent units in GB-RBMs act as on/off switches, providing limited control over the diversity and structure of the hidden space. This bottleneck hampers the model’s ability to disentangle categorical or compositional structure, particularly when the data exhibits rich semantic variation. 2.2 Motivating Discrete Latent Structure: Potts Units In contrast, Potts models, which extend binary variables to q-state categorical variables, offer a compelling mechanism for reintroducing structured discrete latent representations into energy-based models. By allowing each hidden unit to express one of multiple mutually exclusive states, Potts units can represent discrete modes, templates, or symbolic features directly. This increases the combinatorial capacity of the model from 2^{m} (binary) to q^{m} (categorical), enabling richer and more interpretable feature representations. Importantly, Potts units also induce a sharper factorization of latent structure, allowing for the emergence of compositional encodings where different latent units capture orthogonal or complementary concepts. This categorical structure aligns more naturally with real-world semantics and improves both generalization and controllability. 2.3 The Gaussian-Multinoulli RBM (GM-RBM) To address the limitations of GB-RBMs and exploit the representational advantages of Potts variables, we introduce the Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM). In this architecture, the visible layer consists of Gaussian-valued units, modeling real-valued data such as sensor signals or pixel intensities while the hidden layer consists of Multinoulli (Potts) units, each of which selects one of q discrete latent states. Each Potts state corresponds to a different “template” or filter that contributes to the energy of the model, allowing a single hidden unit to encode latent modes or feature classes rather than just presence/absence. The GM-RBM thus combines the smooth representational power of Gaussian inputs with the symbolic richness of discrete latent categories. This hybrid construction restores the model’s ability to capture structured, discrete latent features while retaining flexibility in modeling continuous-valued observations. As we show, the GM-RBM can learn more interpretable latent spaces, support combinatorial recombination of features, and exhibit sharper mode separation in the visible manifold than its Gaussian-Bernoulli counterpart. 3 Theoretical Foundations of the Gaussian-Multinoulli RBM (GM-RBM) The Gaussian-Multinoulli Restricted Boltzmann Machine (GM-RBM) extends the classical Gaussian-Bernoulli RBM by replacing binary latent units with discrete q -state Potts (Multinoulli) variables. This modification introduces a combinatorially structured latent space capable of expressing discrete modes while modeling continuous-valued data in the visible layer. In this section, we develop a mathematical understanding of how this architecture supports richer and more structured representations. 3.1 Energy Function and Architecture The GM-RBM defines a joint energy over visible variables v\in\mathbb{R}^{n} and hidden Potts variables h=(h_{1},\dots,h_{m}) , where each h_{j}\in\{1,\dots,q\} selects a discrete state for hidden unit j : E(v,h)=\frac{1}{2}\sum_{i=1}^{n}(v_{i}-b_{i})^{2}-\sum_{j=1}^{m}c_{j,h_{j}}-\sum_{i=1}^{n}\sum_{j=1}^{m}W_{ij}^{(h_{j})}v_{i} (1) This can

Read the complete text