unveiLing : What Makes Linguistics Olympiad Puzzles Tricky for LLMs?

Mukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, Ikhlasul Akmal Hanif, Daria Kotova, Ekaterina Kochmar, Monojit Choudhury · 2025

Read unveiLing : What Makes Linguistics Olympiad Puzzles Tricky for LLMs? by Mukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, Ikhlasul Akmal Hanif, Daria Kotova, Ekaterina Kochmar, Monojit Choudhury (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-11260.txt.

Introduction

Mukund Choudhary and co-authors probe why linguistics puzzles are hard for language models in a 2025 paper. The puzzles come from Linguistics Olympiad contests and use low-resource languages, so models cannot rely on memorized text. That makes them a clean test of raw language reasoning.

The team labeled 629 problems across 41 languages with features drawn from linguistics. Models struggled most when words had complex internal structure, and did better when a language shared traits with English. Splitting words into their building blocks before solving made puzzles easier for the models. The authors argue that tokenizers, the tools that chop text for models, need language-aware design. The work maps weak spots in how models handle the world's rarer languages.

Catalog record

CREATORMukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, Ikhlasul Akmal Hanif, Daria Kotova, Ekaterina Kochmar, Monojit Choudhury
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-11260

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

unveiLing : What Makes Linguistics Olympiad Puzzles Tricky for LLMs? Mukund Choudhary, KV Aditya Srivatsa, Gaurja Aeron, Antara Raaghavi Bhattacharya, Dang Khoa Dang Dinh, Ikhlasul Akmal Hanif, Daria Kotova, Ekaterina Kochmar, Monojit Choudhury Abstract Large language models (LLMs) have demonstrated potential in reasoning tasks, but their performance on linguistics puzzles remains consistently poor. These puzzles, often derived from Linguistics Olympiad (LO) contests, provide a minimal contamination environment to assess LLMs’ linguistic reasoning abilities across low-resource languages. In this work, we analyze LLMs’ performance on 629 problems across 41 low-resource languages by labelling each with linguistically informed features to unveil weaknesses. Our analyses show that LLMs struggle with puzzles involving higher morphological complexity and perform better on puzzles involving linguistic features that are also found in English. We also show that splitting words into morphemes as a pre-processing step improves solvability, indicating a need for more informed and language-specific tokenisers. These findings thus offer insights into some challenges in linguistic reasoning and modelling of low-resource languages. 1 Introduction Figure 1 : An LO puzzle with unveiLing annotations. In-context data are translation pairs, and problems are to be translated on the basis of the data above. Human proficiency in analysing languages and linguistic reasoning contributes significantly to efficient learning from limited linguistic input. Following frameworks and goals established by Mahowald et al. 2024 ; McClelland et al. 2019 ; Marcus 2020 ; Beguš et al. 2023 , improvements in LLM metalinguistic processing capabilities can enhance general performance across diverse tasks. Linguistics Olympiad (LO) puzzles provide us with rich data to probe into these LLM abilities, especially in extremely low resource languages (LRL), as past research has shown ( Chi et al. 2024 ) . Typically, LO puzzles have parallel data between an unknown language and corresponding forms in a language known to the solver ( Bozhanov & Derzhanski 2013 ) . The solver (typically pre-collegiate level) is expected to deduce the mapping between the linguistic structures of the unknown and the known languages, which involves observing syntactic patterns, semantic correspondences, phonological rules, or a combination of these. We shall refer to them as features akin to linguistic features ( Dryer & Haspelmath 2013 ) . Fig. 1 shows an example of a puzzle set in Bangime (an unknown language) with the corresponding English (a known language) translations. The puzzles are similar to in-context few-shot learning tasks ( Brown et al. 2020 ) where context contains all the data needed to solve the problems . Generally, these are built from data in endangered languages or LRLs to ensure that participants at the Olympiads do not already know the patterns in the data. Therefore, we expect these languages to be absent from the LLM training data as well. This makes LO puzzles a challenging and minimal contamination setting to assess an LLM’s linguistic reasoning abilities, and, for that reason, they have been incorporated into various benchmarks ( Kazemi et al. 2025 ) . Dataset Source Puzzles (Problems) Classes Difficulty Languages modeLing ( Chi et al. 2024 ) Author-made 48 (272) Noun-Adjective (40%), Semantics (10%), Word Order (40%), Possession (10%) 1 (3%), 2 (34%), 3 (23%), 4 (11%), 5 (29%) 19 PuzzLing ( Şahin et al. 2020 ) National Linguistics Olympiads 96 (752) – – 81 LingOly ( Bean et al. 2024 ) UK Linguistics Olympiad 90 (1133) Compounding, Morphology, Numbers, Phonology, Semantics, Syntax Breakthrough (6.5%), Foundation (13.3%), Intermediate (13.5%), Advanced (32.9%), Round 2 (33.7%) 94 Linguini ( Sánchez et al. 2024 ) International Linguistics Olympiad 160 (894) Sequence Transduction, Fill-In Blanks, Number Transliteration – 75 Table 1 : Summary of datasets with Linguistic Olympiad style puzzles. % indicates proportion in data Several researchers have used LO puzzles to study language models’ reasoning abilities (summarized in Tab. 1 ). LLMs (while being impressive in application-based language tasks) perform consistently poorly in these Olympiads, whereas high school students can solve most of the puzzles fully. While these studies show that LLMs struggle, they do not explain why. Since these puzzles reflect the complex interplay of linguistic features in natural languages, pinpointing the causes of poor performance is challenging. On the other hand, a feature-level analysis of LLM weaknesses can enhance our understanding of their linguistic reasoning and can provide insights into bridging the gap between LLM abilities and a core component of human cognitive flexibility to identify patterns, extract rules, and generalise from minimal data across unfamiliar languages. To answer the research question of what linguistic features make LO puzzles challenging for LLMs , we have annotated 64 puzzles (629 problems, 41 LRLs) from existing datasets (Sec. 2.1 ) with 50 linguistic features (from WALS, 1 1 1 World Atlas of Languages by Dryer & Haspelmath 2013 also including other attributes such as similarity to English – see Sec. 2.2 ). We discover that LLM performance (Sec. 2.3 ) is significantly negatively correlated with the number of morphological features of a language present in an LO puzzle (Sec. 3.1 ) and the number of features in a puzzle that have a low coverage of exemplifying data in the puzzle (Sec. 3.3 ). LLM scores also positively correlated with the average number of features in a puzzle that are present in English (Sec. 3.2 ). Further experiments showed that breaking down words in LRL into morphemes in morphologically complex puzzles helped LLMs (Sec. 3.5 ), seemingly indicating the need for more language-specific handling of tokenisation of words in LRL. Findings from the study could help inform future LLM-based research on sample-efficient LRL learning, metalinguistic reasoning, and more. 2 Setting the Stage for unveiLing In this section, we set the stage for unveiling LLM weaknesses in solving LO puzzles by: 1. Curating data and finding unseen puzzles. 2. Establishing an annotation scheme to break down all puzzles using linguistic features. 3. Obtaining LLM responses to evaluate performance against the feature annotations. 2.1 Data Existing Datasets An overview of the four datasets that are based on LO-style puzzles can be found in Tab. 1 . All of these reveal that LLMs are poor at solving LO problems based on Exact-Match Accuracy, which measures the percentage of problems per puzzle where the predicted final answer exactly matches the ground truth. LLMs performed the best (40-60%) in solving puzzles from modeLing ( Chi et al. 2024 ) (author-made, easier LO-style puzzles) and Foundation-level (younger students) puzzles in LingOly ( Bean et al. 2024 ) , while performing the worst (25%) on International Linguistics Olympiad (IOL) puzzles from Linguini ( Sánchez et al. 2024 ) . Filtering out Solvable Puzzles LLMs struggle with Linguini ’s set of IOL puzzles as they are tougher and are designed for participants who have more experience with LO puzzles. Thus we only consider puzzles from modeLing (M) and LingOly (L). To maintain uniformity across datasets and select the simplest tasks, we retained only Rosetta-Stone style ( Bozhanov & Derzhanski 2013 ) puzzles which had paired translations in context and problems that required participants to either translate from an LRL to English or vice versa. This filtering step led to an initial pool of puzzles consisting of all 48 M puzzles 2 2 2 All M puzzles were first solved and checked by the authors for inconsistencies

Read the complete text