Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior
Milad Sefidgaran [email protected], Abdellatif Zaidi [email protected], Piotr Krasnowski [email protected] · 2025
Read Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior by Milad Sefidgaran [email protected], Abdellatif Zaidi [email protected], Piotr Krasnowski [email protected] (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18455.txt.
Introduction
Milad Sefidgaran and co-authors study how a team of watchers, each seeing only one view of a scene, can learn a shared picture without talking to each other. Each client squeezes its own view into a compact code, and a central decoder joins the codes to judge the scene. The puzzle is that no client knows what the others will send.
The team proves bounds on how badly such a system can err on fresh data. The bounds hinge on how much shared structure the codes carry, and they show that copies across views can help rather than hurt. Guided by the math, the team builds a shared prior from a blend of bell curves that clients can learn apart yet use jointly. Tests across data sets and setups back the gains.
Catalog record
| CREATOR | Milad Sefidgaran [email protected], Abdellatif Zaidi [email protected], Piotr Krasnowski [email protected] |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 135 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-18455
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Generalization Guarantees for Multi-View Representation Learning and Application to Regularization via Gaussian Product Mixture Prior Milad Sefidgaran [email protected], Abdellatif Zaidi [email protected], Piotr Krasnowski [email protected] Abstract We study the problem of distributed multi-view representation learning. In this problem, K K agents observe each one distinct, possibly statistically correlated, view and independently extracts from it a suitable representation in a manner that a decoder that gets all K K representations estimates correctly the hidden label. In the absence of any explicit coordination between the agents, a central question is: what should each agent extract from its view that is necessary and sufficient for a correct estimation at the decoder? In this paper, we investigate this question from a generalization error perspective. First, we establish several generalization bounds in terms of the relative entropy between the distribution of the representations extracted from training and “test” datasets and a data-dependent symmetric prior, i.e., the Minimum Description Length (MDL) of the latent variables for all views and training and test datasets. Then, we use the obtained bounds to devise a regularizer; and investigate in depth the question of the selection of a suitable prior. In particular, we show and conduct experiments that illustrate that our data-dependent Gaussian mixture priors with judiciously chosen weights lead to good performance. For single-view settings (i.e., K = 1 K=1 ), our experimental results are shown to outperform existing prior art Variational Information Bottleneck (VIB) and Category-Dependent VIB (CDVIB) approaches. Interestingly, we show that a weighted attention mechanism emerges naturally in this setting. Finally, for the multi-view setting, we show that the selection of the joint prior as a Gaussians product mixture induces a Gaussian mixture marginal prior for each marginal view and implicitly encourages the agents to extract and output redundant features, a finding which is somewhat counter-intuitive. MnLargeSymbols’164 MnLargeSymbols’171 Milad Sefidgaran and Abdellatif Zaidi and Piotr Krasnowski † † heading: 25 2024 1- 22/07 21-0000 † † shortheadings: Generalization Guarantees for Multi-View Representation Learning / Sefidgaran, Zaidi, and Krasnowski † † firstpage: 1 1 Introduction One major problem in learning theory pertains to how to guarantee that a statistical learning algorithm performs on new, unseen, data as well as it does on the used training data, i.e., good generalization properties. This key question, which has roots in various scientific disciplines, has been studied using seemingly unrelated approaches, including compression-based ( Littlestone and Warmuth 1986 ; Blumer et al. 1987 ; Arora et al. 2018 ; Blum and Langford 2003 ; Suzuki et al. 2020 ; Hsu et al. 2021 ; Barsbey et al. 2021 ; Hanneke and Kontorovich 2019 ; Hanneke et al. 2019 ; Bousquet et al. 2020 ; Hanneke and Kontorovich 2021 ; Hanneke et al. 2020 ; Cohen and Kontorovich 2022 ; Sefidgaran et al. 2022 ; Sefidgaran and Zaidi 2024 ) , information-theoretic ( Russo and Zou 2016 ; Xu and Raginsky 2017 ; Steinke and Zakynthinou 2020 ; Esposito et al. 2020 ; Bu et al. 2020 ; Haghifam et al. 2021 ; Neu et al. 2021 ; Aminian et al. 2021 ; Harutyunyan et al. 2021 ; Zhou et al. 2022 ; Lugosi and Neu 2022 ; Hellström and Durisi 2022 ) , PAC-Bayes ( Seeger 2002 ; Langford and Caruana 2001 ; Catoni 2003 ; Maurer 2004 ; Germain et al. 2009 ; Tolstikhin and Seldin 2013 ; Bégin et al. 2016 ; Thiemann et al. 2017 ; Dziugaite and Roy 2017 ; Neyshabur et al. 2018 ; Rivasplata et al. 2020 ; Negrea et al. 2020a ; Negrea et al. 2020b ; Viallard et al. 2021 ) , and intrinsic dimension-based ( Şimşekli et al. 2020 ; Birdal et al. 2021 ; Hodgkinson et al. 2022 ; Lim et al. 2022 ) approaches. In practice, a common approach advocates the usage of a two-part, or encoder-decoder , model, often referred to as representation learning . The encoder part of the model shoots for the extraction of a “minimal” representation of the input (i.e., small generalization error), whereas the decoder part shoots for small empirical risk. One popular such approach is the information bottleneck (IB), which was first introduced in ( Tishby et al. 2000 ) and then extended in various ways ( Shamir et al. 2010 ; Alemi et al. 2017 ; Estella Aguerri and Zaidi 2018 ; Kolchinsky et al. 2019 ; Fischer 2020 ; Rodríguez Gálvez et al. 2020 ; Kleinman et al. 2022 ) . The IB principle is mainly based on the assumption that Shannon’s mutual information between the input and the representation is a good indicator of the generalization error. However, this assumed relationship has been refuted in several works ( Kolchinsky et al. 2018 ; Rodriguez Galvez 2019 ; Amjad and Geiger 2019 ; Geiger and Koch 2019 ; Dubois et al. 2020 ; Lyu et al. 2023 ; Sefidgaran et al. 2023 ) . Rather, recent works ( Blum and Langford 2003 ; Sefidgaran et al. 2023 ) have shown that the generalization error of representation learning algorithms is related to the minimum description length (MDL) of the latent variable and to the so-called geometric compression ( Geiger and Koch 2019 ) . Figure 1 : Distributed multi-view representation learning setup. The approach described thus far involves only a single encoder and a single decoder; and is sometimes loosely referred to as centralized representation learning, in reference to that all training data is available at one place, the encoder. In many real-world scenarios, however, multiple streams of data may be available each at a distinct encoder; every encoder extracts some relevant features from its input data, independently of other encoders; and the extracted features are fused suitably by a decoder in the aim of making a proper decision for the inference task at hand. This setting, which is shown in Fig. 1 and described more formally in Section 2 , is referred to as distributed multi-view representation learning, in reference to that multiple views need to be processed simultaneously by the encoders in a manner that, collectively, the extracted vectors of features enable correct estimation of the label variable by the decoder. We emphasize that in this setting the encoders or clients observe distinct, but possibly statistically correlated views, which are all needed for making inference during the test phase; and, in particular, this differs from setups in which every client has its own independent dataset such as in the popular Federated Learning of ( McMahan et al. 2017 ) . For the distributed multi-view representation learning setting of Fig. 1 , one major difficulty is caused by the encoders not being allowed to interact with each other explicitly. That is, every encoder needs to independently extract a vector of features from its input that is minimal from an MDL perspective and sufficient for estimating the label Y when combined with other extracted vectors of features by other encoders; and, this has to be accomplished without explicit coordination or interaction with those other involved encoders! . In fact, important questions abound in this case. For example: (i) for a better generalization, should the encoders extract redundant or complementary features? and (ii) in supervised learning settings, what regularization induces the encoders to learn, during