Tight PAC-Bayesian Risk Certificates for Contrastive Learning

Anna van Elst, Debarghya Ghoshdastidar · 2024

Read Tight PAC-Bayesian Risk Certificates for Contrastive Learning by Anna van Elst, Debarghya Ghoshdastidar (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03486.txt.

Description

Anna van Elst and Debarghya Ghoshdastidar published this study in 2024. It builds risk certificates for contrastive learning, a way of teaching models from unlabeled data. Such models learn to place similar items close and unlike items apart. The method is widely used, yet its error theory lags behind. Past bounds were either useless in practice or rested on shaky assumptions.

The authors derive PAC-Bayesian certificates that stay meaningful for the popular SimCLR setup. A subtle point drives the work: SimCLR reuses paired samples as negatives for others, which breaks the independence older bounds need. Their bounds fold in that reuse, plus the role of data augmentation and temperature. Tests on a standard image set show far tighter guarantees than before, for both the training loss and later tasks.

Catalog record

CREATORAnna van Elst, Debarghya Ghoshdastidar
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03486

Browse the full library · About the Tower of Babel