BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics

Taj Gillin, Adam Lalani, Kenneth Zhang, Marcel Mateos Salles · 2026

Read BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics by Taj Gillin, Adam Lalani, Kenneth Zhang, Marcel Mateos Salles (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-00366.txt.

Description

Taj Gillin, Adam Lalani, Kenneth Zhang, and Marcel Mateos Salles introduce BERT-JEPA, a training method for BERT-style language models. It adds a predictive embedding objective, drawn from joint embedding predictive architectures, to the usual training. The aim is to fix a collapsed summary-token embedding space and reshape it so that meaning is separated from language. The authors report that this new structure improves results across benchmarks in several languages.

Catalog record

CREATORTaj Gillin, Adam Lalani, Kenneth Zhang, Marcel Mateos Salles
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-00366

Browse the full library · About the Tower of Babel