Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks
Paul Suganthan*, Fedor Moiseev, Le Yan, Junru Wu, Jianmo Ni, Jay Han, Imed Zitouni, Enrique Alfonseca, Xuanhui Wang, Zhe Dong* · 2025
Read Adapting Decoder-Based Language Models for Diverse Encoder Downstream Tasks by Paul Suganthan*, Fedor Moiseev, Le Yan, Junru Wu, Jianmo Ni, Jay Han, Imed Zitouni, Enrique Alfonseca, Xuanhui Wang, Zhe Dong* (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-02656.txt.
Description
Paul Suganthan and co-authors adapt a decoder-based language model, Gemma, into an encoder for tasks that do not generate text, published in 2025. Decoder models have grown huge and powerful, yet encoder models still lead on tasks like classification, regression, and ranking. The one-way structure of decoders limits their direct use for those jobs.
The resulting Gemma Encoder changes the model to read text in both directions. The team systematically tests pooling choices, attention settings, and training values such as dropout. They then benchmark the encoder on a standard language test suite and a ranking benchmark. The results are presented as proof that a strong decoder can be reshaped into a versatile encoder for a wide range of downstream uses.
Catalog record
| CREATOR | Paul Suganthan*, Fedor Moiseev, Le Yan, Junru Wu, Jianmo Ni, Jay Han, Imed Zitouni, Enrique Alfonseca, Xuanhui Wang, Zhe Dong* |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 32 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-02656