AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese
Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins, João Magalhães · 2026
Read AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese by Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins, João Magalhães (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26511.txt.
Introduction
Simplício and co-authors introduce AMALIA, a fully open language model built for European Portuguese. That variant is scarce in training data, and translated tests can miss its local traits. AMALIA uses more high-quality European Portuguese text in both mid-training and post-training stages. The team also releases native benchmarks, with translated tasks plus four new sets on generation, language skill, and bias between variants. AMALIA matches strong rivals on translated tests and clearly beats them on native ones.
Catalog record
| CREATOR | Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins, João Magalhães |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 35437 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.26511
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese Afonso Simplício, Gonçalo Vinagre, Miguel Moura Ramos, Diogo Tavares, Rafael Ferreira, Giuseppe Attanasio, Duarte M. Alves, Inês Calvo, Inês Vieira, Rui Guerra, James Furtado, Beatriz Canaverde, Iago Paulo, Vasco Ramos, Diogo Glória-Silva, Miguel Faria, Marcos Treviso, Daniel Gomes, Pedro Gomes, David Semedo, André Martins, João Magalhães Abstract Despite rapid progress in open large language models (LLMs), European Portuguese (pt-PT) remains underrepresented in both training data and native evaluation, with machine-translated benchmarks likely missing the variant’s linguistic and cultural nuances. We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages. To evaluate pt-PT more faithfully, we release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias. Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations, supporting the case for targeted training and native benchmarking for European Portuguese. 1 1 1 https://github.com/AMALIA-LLM/AMALIA 1 Introduction The rapid development of large language models (LLMs) has fundamentally transformed the field of natural language processing (NLP), with open-weight models 15 ; 65 ; 23 setting new standards in tasks ranging from problem-solving to conversational AI. However, a significant limitation of current state-of-the-art models is their overwhelming focus on English-language data 39 . This imbalance is reflected in evaluation, where the scarcity of native benchmarks leads many researchers to rely on machine translation ( 53 ; 64 , e.g.,) . In turn, such MT benchmarks often fail to capture the linguistic and cultural nuances of content naturally produced by native speakers 43 ; 62 , spurring focused efforts to prioritize native, culturally specific sourcing ( 3 ; 52 , e.g.,) . As a result, many European languages, including European Portuguese, are under-represented in global LLMs, limiting these technologies’ ability to capture the full breadth of Europe’s linguistic and cultural diversity. This paper introduces AMALIA, an LLM designed to address this imbalance by prioritizing European Portuguese and its cultural context during pretraining and post-training. Furthermore, we introduce a set of benchmarks that assess the linguistic performance and cultural sensitivity of LLMs, explicitly tailored for European Portuguese. The process of creating AMALIA began with the collection and processing of large-scale European Portuguese data and subsequent data quality filtering. We leveraged Arquivo.pt, the Portuguese Web Archive, 2 2 2 https://arquivo.pt/ as a primary source for this data. The model was then pretrained on this data and post-trained on specific datasets for instruction following, conversational reasoning, and problem solving. Our experiments reveal that the model is on par with other similarly-sized open models in most machine-translated benchmarks and is superior on the European Portuguese benchmarks. This is a strong indication that LLMs can indeed capture specific traits of underrepresented language varieties, even when the number of examples is orders of magnitude smaller than the dominant language variety and the dominant language, i.e. , Brazilian Portuguese and English, respectively. 2 Related Work The development of fully open large language models has been dominated by English-centric training 39 , although recent efforts have been made in multilingual models 18 ; 33 ; 2 . Despite this, lower resource languages are still under-represented in these models, and language varieties distinction, such as European and Brazilian Portuguese, is still a major challenge, leading to degraded performance on language-specific tasks and cultural contexts. To mitigate this phenomena, several efforts have addressed the lack of data and models for European Portuguese specifically. Gloria 30 introduced an European Portuguese decoder-based LLM pretrained on a large PT-PT text corpora, and Gervasio 50 fine-tuned Llama-Instruct 2 15 for pt-PT. 3 Data Collection and Processing 3.1 Arquivo Data Collections. We collected publicly available data from Arquivo.pt. Following GlórIA’s 30 efforts, we gathered WARC archives comprising a selection of collections that primarily contain general web and free books, thereby improving the variety and quantity of the Portuguese data in the pretraining phase. We obtained this data by copying directly from Arquivo.pt’s data center; in its raw format, it totals 195 terabytes of WARC archives, which were subsequently processed. Filtering. We created a data processing and filtering pipeline inspired by the FineWeb2 42 pipelines, using the Datatrove 41 library. For each data collection, we begin by using URL filtering to remove any ‘.br’ domains to reduce Brazilian Portuguese content. We also use a blacklist to remove sensitive (NSFW) content. Then, we scrape the text from the HTML files using the Trafilatura 5 extractor and perform additional post-scraping by removing any short or duplicate lines in the text. We then followed FineWeb heuristic filters with its parameters calibrated for European Portuguese, namely the Language Identification 24 , FineWeb Quality, and Gopher Repetition and Quality filters 44 . After removing personal data and fixing remaining encoding issues, we deduplicated the data using MinHash 9 , and labeled it according to the EuroFilter quality classification model 33 . Finally, we merged the data in all collections, creating three quality splits (high, medium and low), and we used the high and medium quality, totaling 5.8 billion tokens. GDPR compliance. The data from Arquivo.pt was collected from the public Web and respects an embargo of one year. As mentioned, we removed personal data, namely public IPs, email addresses, and phone numbers. 3.2 Long Context Following 46 , we reused the long documents from EuroLLM’s pretraining corpus 33 , which had previously been truncated to 4 K tokens. We also collected long-context code samples from Stack-v2 32 , considering only repositories with at least 500 stars and 100 forks. Additionally, we included synthetic data focused on improving recall and retrieval in long-context scenarios. Concretely, starting from publicly available datasets, 3 3 3 Datasets available at https://huggingface.co/datasets/lvwerra/needle-llama3-16x8k and https://huggingface.co/datasets/nanotron/needle_32k_finetuning_dataset . we filtered out samples longer than 32 K tokens and removed duplicates. 3.3 Post-train Data To improve AMALIA’s instruction following capabilities, we built a data mixture targeting four categories: instruction following, conversational reasoning, mathematical problem-solving, and safety. This mixture comprises our own synthetic and manually curated data in both European Portuguese and English alongside publicly available datasets from Hugging Face. We only used data sources that had an open-source license associated to it. The complete data mixture is detailed in Table 1 . Our synthetic generation primarily followed the PersonaHub 17 approach to increase diversity in instruction following and conversational scenarios, using the PersonaHub and Nemotron personas 34 to create datasets for general, instruction following, and mathematical tasks. European Portuguese data was produced via machine translation using our pt-PT MT model and Gemma 3-27B 23 , question generation with few-shot prompting, and answer generation with Gemma 3-27B. This choice stems from our observation that Gemma 3-27B was the best available open model for European Portuguese, a finding later corroborated by experimental results (Section 6.4 ). We consistently relied on larger and varied models to produce prompts, questions, and answers in order to reduce the risk of biased data sampling. To ensure high data quality, we applied dataset-specific filtering and formatting strategies. We removed all reasoning traces, eliminated samples containing self-referential content from other models, and utilized the Deita Quality Scorer 29 , which