MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza · 2025

Read MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs by Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-02768.txt.

Description

Jaap Jumelet and three co-authors release in 2025 MultiBLiMP 1.0, a benchmark of linguistic minimal pairs spanning 101 languages. Minimal pairs are sentence twins differing in one grammatical feature, and picking the grammatical twin tests whether a model truly knows the rule. The set covers subject-verb agreement in two forms and holds over 128,000 pairs.

An automated pipeline built the pairs from Universal Dependencies trees and UniMorph tables, keeping the method scalable to new languages. Tested models showed clear weaknesses in low-resource languages. A benchmark of this breadth exposes how far multilingual models still lean on a few rich languages, and gives trainers a yardstick for closing the gap.

Catalog record

CREATORJaap Jumelet, Leonie Weissweiler, Joakim Nivre, Arianna Bisazza
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE58 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-02768

Browse the full library · About the Tower of Babel