ACL: A ligned C ontrastive L earning Improves BERT and Multi-exit BERT Fine-tuning

2026

Read ACL: A ligned C ontrastive L earning Improves BERT and Multi-exit BERT Fine-tuning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03563.txt.

Description

Contrastive learning helps models pull similar examples together, but in supervised fine-tuning it can fight the main classification loss. Liz Li and Wei Zhu investigate that conflict in a 2026 paper on BERT, the classic language encoder. Their pilot experiments show the cross-entropy objective and the contrastive objective often pull in opposite directions.

Their framework, Aligned Contrastive Learning, turns label embeddings into extra augmented samples and aligns them with the matching examples. A gradient rule drops the contrastive term whenever the two objectives clash. A cross-layer version lets a strong deep exit teach the shallow exits of multi-exit BERT. On the GLUE benchmark the approach matches or beats plain fine-tuning and clearly improves multi-exit models' speed-quality trade-off.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE37 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-03563

Browse the full library · About the Tower of Babel