Do LLMs exhibit the same commonsense capabilities across languages?

Iván Martínez-Murillo, Elena Lloret, Paloma Moreda, Albert Gatt · 2025

Read Do LLMs exhibit the same commonsense capabilities across languages? by Iván Martínez-Murillo, Elena Lloret, Paloma Moreda, Albert Gatt (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-06401.txt.

Introduction

Commonsense in a language model is strongest in English and thins in smaller languages, measure Ivan Martínez-Murillo and three co-authors in a 2025 paper. Their benchmark, MULTICOM, extends an existing dataset to English, Spanish, Dutch, and Valencian. The task: generate a commonsensical sentence containing a given triplet of words.

A range of open models, from LLaMA and Qwen to regional entries like Salamandra, are scored by automatic metrics, model judges, and human annotators. English wins consistently, less-resourced languages lag significantly, and adding context gives mixed help that tends to favor the underrepresented languages. The dataset is released publicly, and the authors frame the gap as a standing limitation of multilingual generation.

Catalog record

CREATORIván Martínez-Murillo, Elena Lloret, Paloma Moreda, Albert Gatt
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-06401

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Do LLMs exhibit the same commonsense capabilities across languages? Iván Martínez-Murillo, Elena Lloret, Paloma Moreda, Albert Gatt Abstract This paper explores the multilingual commonsense generation abilities of Large Language Models (LLMs). To facilitate this investigation, we introduce MULTICOM, a novel benchmark that extends the COCOTEROS dataset to four languages: English, Spanish, Dutch, and Valencian. The task involves generating a commonsensical sentence that includes a given triplet of words. We evaluate a range of open-source LLMs, including LLaMA, Qwen, Gemma, EuroLLM, and Salamandra, on this benchmark. Our evaluation combines automatic metrics, LLM-as-a-judge approaches (using Prometheus and JudgeLM), and human annotations. Results consistently show superior performance in English, with significantly lower performance in less-resourced languages. While contextual support yields mixed results, it tends to benefit underrepresented languages. These findings underscore the current limitations of LLMs in multilingual commonsense generation. The dataset is publicly available in https://huggingface.co/datasets/gplsi/MULTICOM . 1 Introduction Commonsense reasoning is a fundamental human ability that allows individuals to make inferences without needing everything to be explicitly stated Do et al. 2024 . For example, people intuitively know that a dog can bark but not meow, or that a pencil is typically used for writing or drawing. Transferring this type of cognitive skill to Artificial Intelligence (AI) systems is challenging, as it involves enabling models to make inferences beyond what is directly expressed. LLMs have become the state of the art in Natural Language Generation (NLG) due to their training on vast amounts of text that capture a wide range of world knowledge. As a result, they are able to exhibit at least a superficial level of commonsense, thanks to the information encoded in their parameters. However, while LLMs are trained on multilingual data, the majority of that data is in English. This imbalance can lead to disparities in performance across languages Blasi et al. 2022 ; Xu et al. 2024 ; Qin et al. 2025 . In this paper, we investigate whether LLMs demonstrate the same level of ability to generate text which conforms to commonsense intuitions across different languages. To explore this topic, we use a constrained commonsense generation task, using M ulti C om , a novel dataset inspired by the CommonGen benchmark Lin et al. 2020 : given a triplet of concepts, the model must generate a coherent and plausible sentence that connects them. Additionally, we examine whether providing a supporting context in addition to the concept triple improves the model’s ability to produce meaningful outputs. We evaluate the outcomes across four different languages using automatic metrics, LLM-as-judge evaluations, and human judgments. Our results show a significant disparity across different languages in the extent to which LLMs are able to generate texts compatible with commonsense. In summary, our contributions are as follows: (i) We present M ulti C om , a novel benchmark dataset for commonsense generation, which extends an existing dataset to encompass four different languages (English, Dutch, Spanish and Valencian); (ii) We introduce and study the role of supporting linguistic context as a way to assist the model during the generation to provide more accurate texts and; (iii) We perform a comprehensive evaluation of commonsense generation capabilities across five large language model families, varying in both size and configuration. 2 Related Work Commonsense Generation Commonsense knowledge refers to the information that people typically infer in everyday situations, even when it is not explicitly stated Bhargava and Ng 2022 . Incorporating this type of knowledge into NLG systems has been recognised as both important and challenging Wang et al. 2021 , motivating several several benchmarks and shared tasks. The Avicenna task Aghahadi and Talebpour 2022 focuses on generating a conclusion that logically connects two sentences with a syllogistic relationship. Winograd Schemas test models’ ability to resolve referential ambiguities which require commonense reasoning to arrive at the most likely interpretation in a given context Levesque et al. 2012 . Other research has concentrated on more constrained generation tasks, such as keyword-to-text. For instance, the SituatedGen dataset Zhang and Wan 2024 involves generating a pair of contrasting sentences based on a set of concepts, including temporal or geographical entities. Similarly, CommonGen Lin et al. 2020 and C 2 Gen Carlsson et al. 2022 require generating a coherent and plausible sentence that describes an everyday scenario using a given set of keywords. C 2 Gen also incorporates contextual information to improve the relevance and accuracy of the generated text. Despite these advances, most research in this area has focused on English. To address this limitation, the COCOTEROS corpus Maestre et al. 2024 was introduced to support commonsense text generation in Spanish. Based on the same principles as CommonGen and C 2 Gen, COCOTEROS aims to generate plausible sentences in Spanish given a set of input concepts. Nevertheless, there is still a need to evaluate commonsense reasoning across different languages, especially with the emergence of large language models (LLMs) capable of multilingual text generation. The present paper addresses this gap by evaluating the multilingual commonsense generation abilities of LLMs. Multilinguality in LLMs Multilinguality is a key feature of LLMs, as they are trained on massive datasets containing texts in multiple languages. This enables them to understand and generate text across a wide range of languages. As a result, LLMs are often able to produce high-quality outputs in various languages. Nevertheless, recent research suggests that LLM capabilities are not uniform across languages, while many languages remain under-represented in training data Blasi et al. 2022 ; Xu et al. 2024 ; Qin et al. 2025 . Several studies have assessed the performance of LLMs across different languages and tasks. For example, Zhang et al. 2023 examine ChatGPT’s performance on three NLP tasks and find that it shows a strong bias toward English, which can hinder its effectiveness in other languages. Similarly, Xuan et al. 2025 introduce MMLU-ProX, a multilingual benchmark derived from English, and report significant performance disparities across languages. Thellmann et al. 2024 also contribute to this line of research by proposing a benchmark based on translated datasets covering several European languages. In the specific case of constrained, commonsense generation tasks, there has been very limited cross-lingual evaluation, in part due to the paucity of resources. This motivates our development of M ulti C om . 3 The M ulti C om dataset We introduce M ulti C om , a new benchmark to test the capabilities of generative models in commonsense generation tasks. Similarly to CommonGen Lin et al. 2020 , we frame the task in terms of generating text from a triple of keywords or concepts. However, M ulti C om further incorporates contextual support, and crucially covers multiple languages (English, Spanish, Dutch and Valencian), including low-resourced languages. 3.1 Data As our starting point, we use the COCOTEROS dataset Maestre et al. 2024 , which was specifically created to assess commonsense generation in Spanish. Each instance in the COCOTEROS dataset includes three components: • Keywords: a set of three words that should appear in a generated sentence. • Context: a group of sentences that provide semantic or thematic background to support the inclusion of the keywords. • Target sentence: a grammatically correct and commonsense-consistent sentence that includes all three keywords or suitable variations. M ulti C om extends COCOTEROS

Read the complete text