ACL: A ligned C ontrastive L earning Improves BERT and Multi-exit BERT Fine-tuning
2026
Read ACL: A ligned C ontrastive L earning Improves BERT and Multi-exit BERT Fine-tuning (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-03563.txt.
Description
Contrastive learning helps models pull similar examples together, but in supervised fine-tuning it can fight the main classification loss. Liz Li and Wei Zhu investigate that conflict in a 2026 paper on BERT, the classic language encoder. Their pilot experiments show the cross-entropy objective and the contrastive objective often pull in opposite directions.
Their framework, Aligned Contrastive Learning, turns label embeddings into extra augmented samples and aligns them with the matching examples. A gradient rule drops the contrastive term whenever the two objectives clash. A cross-layer version lets a strong deep exit teach the shallow exits of multi-exit BERT. On the GLUE benchmark the approach matches or beats plain fine-tuning and clearly improves multi-exit models' speed-quality trade-off.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-03563