Tight PAC-Bayesian Risk Certificates for Contrastive Learning
Anna van Elst, Debarghya Ghoshdastidar · 2024
Read Tight PAC-Bayesian Risk Certificates for Contrastive Learning by Anna van Elst, Debarghya Ghoshdastidar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03486.txt.
Description
Anna van Elst and Debarghya Ghoshdastidar published this study in 2024. It builds risk certificates for contrastive learning, a way of teaching models from unlabeled data. Such models learn to place similar items close and unlike items apart. The method is widely used, yet its error theory lags behind. Past bounds were either useless in practice or rested on shaky assumptions.
The authors derive PAC-Bayesian certificates that stay meaningful for the popular SimCLR setup. A subtle point drives the work: SimCLR reuses paired samples as negatives for others, which breaks the independence older bounds need. Their bounds fold in that reuse, plus the role of data augmentation and temperature. Tests on a standard image set show far tighter guarantees than before, for both the training loss and later tasks.
Catalog record
| CREATOR | Anna van Elst, Debarghya Ghoshdastidar |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 89 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-03486