Supervised and Unsupervised protocols for hetero-associative neural networks
Andrea Alessandrelli, Adriano Barra, Andrea Ladiana, Andrea Lepre, Federico Ricci-Tersenghi · 2025
Read Supervised and Unsupervised protocols for hetero-associative neural networks by Andrea Alessandrelli, Adriano Barra, Andrea Ladiana, Andrea Lepre, Federico Ricci-Tersenghi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18796.txt.
Introduction
Alessandrelli and co-authors extend a classic memory network to juggle triples. Their three-directional associative memory links three layers of binary neurons with generalized Hebbian couplings, so it can store and retrieve structured triplets of patterns rather than single items. The paper builds learning rules for two regimes: supervised, with a teacher showing the right triples, and unsupervised, where the network finds structure on its own.
Using replica theory and interpolation methods from the statistical mechanics of disordered systems, the authors derive equations for how much data the network can hold and how well it retrieves. Simulations back the theory, including on structured data. The work highlights how layers cooperate, not just neurons, which may inform future network design.
Catalog record
| CREATOR | Andrea Alessandrelli, Adriano Barra, Andrea Ladiana, Andrea Lepre, Federico Ricci-Tersenghi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 168 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18796
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Supervised and Unsupervised protocols for hetero-associative neural networks Andrea Alessandrelli, Adriano Barra, Andrea Ladiana, Andrea Lepre, Federico Ricci-Tersenghi Abstract This paper introduces a learning framework for Three-Directional Associative Memory (TAM) models, extending the classical Hebbian paradigm to both supervised and unsupervised protocols within an hetero-associative setting. These neural networks consist of three interconnected layers of binary neurons interacting via generalized Hebbian synaptic couplings that allow learning, storage and retrieval of structured triplets of patterns. By relying upon glassy statistical mechanical techniques (mainly replica theory and Guerra interpolation), we analyze the emergent computational properties of these networks, at work with random (Rademacher) datasets and at the replica-symmetric level of description: we obtain a set of self-consistency equations for the order parameters that quantify the critical dataset sizes (i.e. their thresholds for learning) and describe the retrieval performance of these networks, highlighting the differences between supervised and unsupervised protocols. Numerical simulations validate our theoretical findings and demonstrate the robustness of the captured picture about TAMs also at work with structured datasets. In particular, this study provides insights into the cooperative interplay of layers, beyond that of the neurons within the layers, with potential implications for optimal design of artificial neural network architectures. 1 Introduction The attractor dynamics inherent in Hopfield models serves as a cornerstone for understanding how neural networks store and retrieve patterns of information. In classical Hopfield systems, if the load and the noise are not too high, each attractor represents a stable state toward which the network converges despite the presence of perturbations. This concept is not only central to artificial neural computation but also finds significant parallels in biological processes Szedlak et al., 2014 ; Szedlak et al., 2017 . While Hopfield networks excel at recalling individual patterns through energy minimization Amit, 1992 , real-world cognition often requires integrating distinct data streams—a capability better captured by hetero-associative models Kosko, 1988 . Hetero-associative neural networks, indeed, extend the principles of auto-associative memory models Amit et al., 1985 ; Amit et al., 1985a ; Amit, 1992 ; Coolen et al., 2005 , such as the Hopfield network Hopfield, 1982 , by enabling associations between distinct patterns rather than reinforcing individual ones. Traditional hetero-associative networks, namely the Bidirectional Associative Memories (BAM) introduced by Kosko in the eighties, operate by establishing pairwise connections between two neuron layers, encoding input-output mappings via Hebbian-like synaptic updates Ackley et al., 1985 : once one of its two layers is provided with a noisy pattern, not only that layer performs pattern recognition returning the denoised input but, also, the other layer retrieves the conjugate pattern of the presented one. While effective in simple pattern association tasks, these architectures lack the capacity to process higher-order relational structures, which are crucial in domains such as cognitive modeling, language processing and hierarchical memory retrieval Mézard et al., 1987 . A natural extension, the Three-directional Associative Memory (TAM), overcomes this limitation by introducing a tripartite network where three layers interact via generalized Hebbian couplings, enabling the storage and retrieval of triplets of patterns, that may give rise to complex structures 1 1 1 We stress that several composite signals are built by triplets of elementary inputs as e.g. in audio signal processing (where chords are built of by three notes in their simplest setting) or in video signal processing (where the primary colors are three). Albanese et al., 2022 . This tripartite architecture outperforms w.r.t. bipartite systems in learning, storing and retrieving structured triplets of patterns, thus facilitating advanced tasks such as pattern disentanglement and frequency modulation Agliari et al., 2025 ; Agliari et al., 2025b . Recent advancements in the statistical physics of neural networks further justify the need for extending associative memory models beyond bidirectional constraints, as such extensions improve robustness and versatility Barbier et al., 2024 ; Barbier et al., 2025 ; Fachechi et al., 2025 ; Alessandrelli et al., 2025 . From a neurobiological standpoint, the functional properties of the TAM model can be understood in light of two complementary mechanisms observed in the hippocampus: pattern completion and pattern separation Kandel et al., 2000 . The CA3 subregion relies on strong recurrent connectivity to reactivate full stored patterns from minimal cues, mirroring the TAM’s capability to reconstruct entire configurations from incomplete inputs. Conversely, the dentate gyrus, via a process often described as “expansion recoding”, is believed to map partially overlapping or noisy inputs onto sufficiently distinct neural representations, thus reducing interference and improving discriminability—an ability that the TAM architecture leverages to disentangle similar patterns. By unifying these two processes, the TAM model aligns with core principles of hippocampal function, thereby enabling robust and biologically inspired pattern recognition and pattern disentanglement. While in Agliari et al., 2025 ; Agliari et al., 2025b we introduced the TAM and inspected its retrieval capabilities as pattern recognition and pattern disentanglement, the core contribution of the present study is the systematic examination of supervised and unsupervised hetero-associative Hebbian learning protocols , i.e. the exploration of weight’s dynamics within the TAM networks and the consequent collective acquired retrieval capabilities by their neurons. Supervised learning —characterized by explicit labeling of input patterns— allows the network to disambiguate overlapping information through externally provided guidance Alemanno et al., 2023 ; Franz et al., 2001 ; Agliari et al., 2020a ; Talagrand, 2003 . In contrast, unsupervised learning relies solely on statistical correlations among observed patterns, necessitating the emergence of self-organized feature representations Agliari et al., 2022 ; Coolen et al., 2025 ; Agliari et al., 2025a . Prior works on associative memories have demonstrated that learning efficacy and retrieval quality strongly depend on the interplay between dataset structure and synaptic plasticity Zdeborova, 2020 ; Goldt et al., 2020 ; Marino & Ricci-Tersenghi, 2024 ; Barra et al., 2023 ; Negri et al., 2023 ; Roudi et al., 2025 . We extend these insights to TAM by deriving self-consistency equations for the order parameters of the network that capture the model’s macroscopic behavior under varying degrees of supervision as the control parameters are made to vary. Methodologically, our approach integrates analytical techniques from statistical mechanics of spin glasses, including replica theory Mézard et al., 1987 ; Amit, 1992 ; Agliari et al., 2020 ; Barra et al., 2010 ; Huang, 2017 and Guerra’s interpolation method Guerra, 2001 ; Barra et al., 2010a ; Barra et al., 2011 ; Agliari et al., 2020 , to achieve an exact expression for the free energy -and, thus, self-consistencies and computational phase transitions- of TAM networks at the fairly standard replica symmetric level of description Amit, 1992 ; Coolen et al., 2005 . These theoretical results are complemented by Monte Carlo simulations, which validate the predicted performance across different training scenarios. Notably, our findings highlight the emergence of cooperative dynamics between layers, wherein dataset quality disparities influence retrieval capabilities in a non-trivial manner. Specifically, layers trained with low-entropy datasets benefit from interactions with layers receiving higher-quality inputs, leading to an overall enhancement of network-wide retrieval performance. This phenomenon, termed “ cooperativeness ”, challenges conventional assumptions regarding the independence of memory layers in associative networks Shwartz-Ziv & Tishby, 2017 ; Alessandrelli et al., 2025 . Beyond theoretical implications, our work has practical relevance for artificial neural network architectures: the ability to