ChocoLlama: Lessons Learned From Teaching Llamas Dutch
Matthieu Meeus, Anthony Rathé, François Remy, Pieter Delobelle, Jens-Joris Decorte Thomas Demeester, [10pt] Imperial College London Cavell Ghent University, [10pt] Aleph Alpha imec TechWolf · 2024
Read ChocoLlama: Lessons Learned From Teaching Llamas Dutch by Matthieu Meeus, Anthony Rathé, François Remy, Pieter Delobelle, Jens-Joris Decorte Thomas Demeester, [10pt] Imperial College London Cavell Ghent University, [10pt] Aleph Alpha imec TechWolf (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-07633.txt.
Description
Matthieu Meeus and co-authors report lessons from teaching the Llama language models to work in Dutch. Dutch is spoken by about thirty million people, yet it is thin in the data used to train large models. The team built ChocoLlama by adapting Llama-2 and Llama-3 with a large body of Dutch text.
Their 2024 paper describes continued pretraining with low-rank adaptation, plus Dutch posttraining steps drawn from prior work. For Llama-2 they tested both the stock tokenizer and a new Dutch one with reset embeddings. They scored the models on standard tests and on a new Dutch benchmark. The results show that low-rank adaptation scales well for language adaptation, and that a new tokenizer can help when weights are reset with care.
Catalog record
| CREATOR | Matthieu Meeus, Anthony Rathé, François Remy, Pieter Delobelle, Jens-Joris Decorte Thomas Demeester, [10pt] Imperial College London Cavell Ghent University, [10pt] Aleph Alpha imec TechWolf |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-07633