ESNLIR: A Spanish Multi-Genre Dataset with Causal Relationships
Johan Rodríguez, Nicolás Perez, Rubén Manrique · 2025
Read ESNLIR: A Spanish Multi-Genre Dataset with Causal Relationships by Johan Rodríguez, Nicolás Perez, Rubén Manrique (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-08803.txt.
Introduction
Researchers built a Spanish dataset for natural language inference in this 2025 paper. The task asks a system to judge how two pieces of text relate, for example whether one follows from the other. Work on it is rich for English but thin for Spanish.
Their dataset, ESNLIR, spans many genres of Spanish text. It pays special care to causal links, where one event brings about another. The team also set a first baseline for the task, using models from the BERT family. Their findings show that a wider mix of genres helps a model generalize to new text. Code, notebooks, and the full data of the study are posted in a public archive. The work gives Spanish language technology a resource it lacked.
Catalog record
| CREATOR | Johan Rodríguez, Nicolás Perez, Rubén Manrique |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 37 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-08803
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ESNLIR: A Spanish Multi-Genre Dataset with Causal Relationships Johan Rodríguez, Nicolás Perez, Rubén Manrique Abstract Natural Language Inference (NLI), also known as Recognizing Textual Entailment (RTE), serves as a crucial area within the domain of Natural Language Processing (NLP). This area fundamentally empowers machines to discern semantic relationships between assorted sections of text. Even though considerable work has been executed for the English language, it has been observed that efforts for the S panish language are relatively sparse. Keeping this in view, this paper focuses on generating a multi-genre Spanish dataset for NLI , ESNLIR , particularly accounting for causal R elationships. A preliminary baseline has been conceptualized and subjected to an evaluation, leveraging models drawn from the BERT family. The findings signify that the enrichment of genres essentially contributes to the enrichment of the model’s capability to generalize. The code, notebooks and whole datasets for this experiments is available at: https://zenodo.org/records/15002575 . If you are interested only in the dataset you can find it here: https://zenodo.org/records/15002371 . 1 Introduction In the field of Natural Language Processing, several applications including Information Retrieval (IR), Question Answering (QA), and Information Extraction (IE) necessitate machine comprehension of semantic meaning in text corpora Dagan et al. (2006) . However, understanding isolated sentences is insufficient in these contexts. Being able to grasp the relationships between sentences is equally imperative. Consequently, Natural Language Inference (NLI) or Recognizing Textual Entailment (RTE) has been developed to ascertain whether one sentence, the hypothesis, can be inferred from a different sentence, the premise Dagan et al. (2006) . Nonetheless, a binary label proved inadequate for encompassing all possible semantic relationships between sentences, such as contradiction Giampiccolo et al. (2008) . Hence, a three-way labeling scheme was devised, which comprises of Entailment , Neutral and Contradiction classes. This labeling technique has been employed in the development of various contemporary benchmarks such as RTE4 Giampiccolo et al. (2008) , SICK Marelli et al. (2014) , which are small datasets with less than 10,000 examples, SNLI Bowman et al. (2015) , the first large scale dataset with more than 500,000 examples, MNLI Williams et al. (2018) , the multi-genre version of SNLI, and XNLI Conneau et al. (2018) , a multilingual subset of XNLI. While commendable classification performance has been achieved on these benchmarks, the sciNLI Sadat and Caragea (2022) benchmark underlines the need for a novel label, Reasoning , to encapsulate causal relationships in genres such as scientific text, where ideas often manifest as cause-effect sequences. The same benchmark introduces a method for automated premise-hypothesis pair extraction, diverging from the standard practice of requiring human annotation for sentence generation, thus facilitating the extraction of large volumes of examples without human supervision. This method has also been successfully employed in the msciNLI benchmark Sadat and Caragea (2024) for multi-genre configurations. In particular, these works Sadat and Caragea (2022) ; Sadat and Caragea (2024) state that linking phrases between sentences are a consequence of their type of semantic relationship, and that some linking phrases such as "thus" and "therefore" show cause-effect relationships between sentences that most previous works did not take into account. However, limited research has been executed on NLI in Spanish, with the exception of the Spanish corpus included in XNLI. The first benchmark in this area was created by generating sentence templates from a Spanish Question Answering corpus Peñas et al. (2006) but is relatively small having 2,962 examples and adopts a binary label for Entailment . Despite being subjected to traditional models such as Support Vector Machines, the SPARTE benchmark achieved an accuracy of 60.83 Castillo (2010) . Another benchmark, INFERES Kovatchev and Taulé (2022) , produced with human annotation and exhibiting a three-way labeling technique, scans across six domains extracted from Wikipedia. It is rather more extensive with 8,055 examples and includes a Contradiction category. Other studies Kauchak et al. (2019) have indicated that viable results can be attained in medical domain text by employing linking phrases to extract sentence pairs and ensuing classification of sentence relationships. Additionally, certain model-based approaches like SILT Huertas-Tato et al. (2022) and Facter-Check Martín et al. (2022) have demonstrated that BERT-based models can achieve good classification performance. However, these models may unintentionally learn heuristics from annotation artifacts McCoy et al. (2019) and may even predict labels by merely examining the hypothesis Gururangan et al. (2018) . It is apparent from the existing Spanish NLI research that there is significant room for improvement. One avenue could be the creation of a larger benchmark to facilitate the training of expansive deep learning architectures. Ideally, such a benchmark would span multiple genres, thereby encompassing various writing styles and multitudinous topics to enhance model generalization, akin to MNLI and MSciNLI. Further improvements could be made by including a label to account for causal relationships, as exhibited in sciNLI and MSciNLI. Not least, model evaluations should seek to measure vulnerability to learning artificial patterns from the annotation artifacts in the benchmark in addition to conventional classification metrics Gururangan et al. (2018) Naik et al. (2018) . In light of these considerations, we propose a novel Spanish NLI multi-genre dataset, ESNLIR , Reasoning-Spanish-Natural-Language-Inference dataset, the extraction methodology for which is based on sciNLI. We apply BERT-based models to corpora of multiple genres to establish a baseline of not only classification metrics but also a method for detecting potential annotation and model artifacts. With this in mind, this paper first defines the dataset extraction, followed by the training and evaluation results for the baseline and BERT-based models, including insights into the model annotation artefacts and stress tests. 2 Dataset The data collection process for ESNLIR is akin to that outlined in sciNLI, focusing on searching for linking phrases within extensive corpora. This approach allows for the automatic extraction of premise-hypothesis pairs, accompanied by a brief label validation procedure to ensure a basic standard of data quality. In this section, we provide a detailed description of this process, covering the sources of corpora, genres, and cardinality. 2.1 Multi-Genre Corpora Following the hypothesis from the MNLI dataset, which suggests that incorporating multiple genres enhances models’ generalization capabilities, we selected 34 Spanish corpora to represent eight distinct writing genres: • articles: Web articles and encyclopedia articles. • books: Literature in general, consisting of open access or public domain books. • comments: Web comments on social networks, accounting for non-formal texts. • legal: Legal documents from Colombia. • clinical: Clinical cases extracted from open source datasets. • news: News articles extracted from multiple sources. • talks: Speeches extracted from TED, which extend the non-formal texts group. • theses: Theses extracted from the website of the Universidad de Los Andes, Colombia, with access for academic purposes. The domain and genre of each selected corpora can be found in Table 1 . 2.2 Premise-hypothesis extraction For each corpus, an identical process is used to extract premise-hypothesis pairs. Using a methodology similar to sciNLI, certain linking phrases are identified as indicators of various semantic relationships between sentences, as demonstrated in Table 2 . Based on these criteria, for each document in the corpus, sentences starting with any of the linking phrases listed in Table 2 are identified and used as hypotheses in sentence pairs. To construct these