HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench

Yueyang Wang, Jiawei Fu, Baolong Bi, Xili Wang, Xiaoqing Liu · 2026

Read HE-SNR: Uncovering Latent Logic via Entropy for Guiding Mid-Training on SWE-bench by Yueyang Wang, Jiawei Fu, Baolong Bi, Xili Wang, Xiaoqing Liu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-20255.txt.

Description

Yueyang Wang and four colleagues propose HE-SNR, a way to guide the mid-training of large language models for software tasks. SWE-bench is the standard test of model skill on real coding problems. The skills it tests are mostly built during mid-training, then drawn out by later fine-tuning.

The authors argue that good metrics to steer mid-training are missing. Their approach uses entropy to uncover latent logic in training data, and a signal-to-noise view to judge its value. Better mid-training guidance could raise coding skill without simply adding more data or compute.

Catalog record

CREATORYueyang Wang, Jiawei Fu, Baolong Bi, Xili Wang, Xiaoqing Liu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-20255

Browse the full library · About the Tower of Babel