ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs

Inês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães · 2026

Read ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs by Inês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26516.txt.

Description

Vieira and co-authors introduce ALBA, a benchmark for language skill in European Portuguese. Most data and tests for Portuguese lean on the Brazilian variant, leaving the European one underserved. ALBA was built by hand by language experts to probe eight areas, from syntax and word play to culture-bound meaning. A language-model judge makes scoring of generated text scalable. Tests across many models show uneven skill across the eight areas, which the authors say calls for variety-aware testing.

Catalog record

CREATORInês Vieira, Inês Calvo, Iago Paulo, James Furtado, Rafael Ferreira, Diogo Tavares, Diogo Glória-Silva, David Semedo, João Magalhães
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33407
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26516

Browse the full library · About the Tower of Babel