KletterMix: Climbing Toward High-Quality German Pretraining Data

Maurice Kraus 1,2 Ruben Härle 1,2 Sebastian Sztwiertnia 1,2 Abbas Goher Khan 4 Mehdi Ali 3,4 Michael Fromm 3,4 Kristian Kersting 1,5,6,7 1 AI & ML Group, TU Darmstadt 2 Lab1141 3 Lamarr Institute 4 Fraunhofer IAIS 5 hessian.AI 6 German Research Center for AI (DFKI) 7 Centre for Cognitive Science, TU Darmstadt {maurice.kraus,ruben.haerle}@tu-darmstadt.de · 2026

Read KletterMix: Climbing Toward High-Quality German Pretraining Data by Maurice Kraus 1,2 Ruben Härle 1,2 Sebastian Sztwiertnia 1,2 Abbas Goher Khan 4 Mehdi Ali 3,4 Michael Fromm 3,4 Kristian Kersting 1,5,6,7 1 AI & ML Group, TU Darmstadt 2 Lab1141 3 Lamarr Institute 4 Fraunhofer IAIS 5 hessian.AI 6 German Research Center for AI (DFKI) 7 Centre for Cognitive Science, TU Darmstadt {maurice.kraus,ruben.haerle}@tu-darmstadt.de (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-03773.txt.

Description

Kraus and co-authors release KletterMix, a German corpus for pretraining language models. Top pretraining data drives modern model quality. German resources lag far behind English ones in size, curation, and documentation. Few are validated by controlled training runs. KletterMix is built by translating a leading English corpus into German. Document bounds, metadata, and topic mix are preserved.

Translation quality checks use COMETKiwi across many domains. Scores stay strong, keeping much of the source's semantic and stylistic richness. Corpus analyses cover length spread, topics, sources, and geography. Controlled pretraining and annealing runs test the data as fuel. Models trained on KletterMix beat those trained on rival German corpora on German tasks. The gains show curated translation can lift a whole language's data ecosystem. The corpus is designed as a reusable artifact for the community.

Catalog record

CREATORMaurice Kraus 1,2 Ruben Härle 1,2 Sebastian Sztwiertnia 1,2 Abbas Goher Khan 4 Mehdi Ali 3,4 Michael Fromm 3,4 Kristian Kersting 1,5,6,7 1 AI & ML Group, TU Darmstadt 2 Lab1141 3 Lamarr Institute 4 Fraunhofer IAIS 5 hessian.AI 6 German Research Center for AI (DFKI) 7 Centre for Cognitive Science, TU Darmstadt {maurice.kraus,ruben.haerle}@tu-darmstadt.de
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE79 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.03773

Browse the full library · About the Tower of Babel