Global Minimizers of Sigmoid Contrastive Loss

Kiril Bangachev, Guy Bresler, Iliyas Noman, Yury Polyanskiy · 2025

Read Global Minimizers of Sigmoid Contrastive Loss by Kiril Bangachev, Guy Bresler, Iliyas Noman, Yury Polyanskiy (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-18552.txt.

Description

Contrastive training teaches models to place matching items close and others apart. Kiril Bangachev, Guy Bresler, Iliyas Noman, and Yury Polyanskiy study the best possible outcomes of the sigmoid version of that training in this 2025 paper. The sigmoid loss powers recent image-text models such as SigLIP. Two knobs, temperature and bias, can be trained along with the model.

The team shows those knobs can drive the loss to zero for a rich family of layouts they call constellations. Each constellation is set by a margin and a relative bias. The layouts link to spherical codes, a classic packing problem. The theory explains SigLIP's strong retrieval scores and the gap seen between image and text clouds. It also gives the dimension needed for high-quality representations. A reworked loss with explicit relative bias trained better in synthetic tests.

Catalog record

CREATORKiril Bangachev, Guy Bresler, Iliyas Noman, Yury Polyanskiy
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE90 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-18552

Browse the full library · About the Tower of Babel