Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction

L. Ghiringhelli, A. Zambon, G. Tiana · 2026

Read Sampling at intermediate temperatures is optimal for training large language models in protein structure prediction by L. Ghiringhelli, A. Zambon, G. Tiana (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-29529.txt.

Description

Ghiringhelli and co-authors probe how transformer models learn protein structure from sequences. They sample the loss landscape at varied temperatures with Langevin dynamics. Unlike non-attention networks, transformers show no sharp first-order-like transition in loss. That leaves a band of mid-range temperatures with good learning traits. At those temperatures, most layer weights stay highly conserved. The team gives a practical way to pick the embedding size. Models sampled there also predict mutation energy shifts better than standard training.

Catalog record

CREATORL. Ghiringhelli, A. Zambon, G. Tiana
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62338
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.29529

Browse the full library · About the Tower of Babel