Long-Context Encoder Models for Polish Language Understanding
Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta · 2026
Read Long-Context Encoder Models for Polish Language Understanding by Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12191.txt.
Description
Dadas and co-authors give Polish a long-context encoder model that reads up to 8192 tokens. Decoder-only models dominate recent NLP, yet encoders stay cheap and lean for classification-style tasks. Classic encoders read only short windows, which blocks long documents. Training first adapts positional embeddings, then continues pre-training over all parameters. Smaller compressed variants come from knowledge distillation for tight budgets. Across 25 tasks the model posts the best average among Polish and multilingual rivals, with clear wins on long texts.
Catalog record
| CREATOR | Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 31793 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12191