Long-Context Encoder Models for Polish Language Understanding

Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta · 2026

Read Long-Context Encoder Models for Polish Language Understanding by Sławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12191.txt.

Description

Dadas and co-authors give Polish a long-context encoder model that reads up to 8192 tokens. Decoder-only models dominate recent NLP, yet encoders stay cheap and lean for classification-style tasks. Classic encoders read only short windows, which blocks long documents. Training first adapts positional embeddings, then continues pre-training over all parameters. Smaller compressed variants come from knowledge distillation for tight budgets. Across 25 tasks the model posts the best average among Polish and multilingual rivals, with clear wins on long texts.

Catalog record

CREATORSławomir Dadas, Rafał Poświata, Marek Kozłowski, Małgorzata Grębowiec, Michał Perełkiewicz, Paweł Klimiuk, Przemysław Boruta
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31793
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.12191

Browse the full library · About the Tower of Babel