Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations

Emil Nuutinen, Iiro Rastas, Filip Ginter · 2025

Read Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations by Emil Nuutinen, Iiro Rastas, Filip Ginter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-05963.txt.

Introduction

Emil Nuutinen, Iiro Rastas, and Filip Ginter present a 2025 paper on a simple way to translate question answering data sets into new languages. Such sets mark the exact span of text that answers each question. Translating the text while keeping the spans aligned is normally fiddly work.

The team used a commercial translator's document feature, which keeps formatting marks in place. With it they built a Finnish version of a large English question set. They judged the result four ways: direct checks, comparison with rival sets, a round trip translation test, and the scores of models trained on it. The simple route beat more complex ones in each check.

Catalog record

CREATOREmil Nuutinen, Iiro Rastas, Filip Ginter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2501-05963

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Finnish SQuAD: A Simple Approach to Machine Translation of Span Annotations Emil Nuutinen, Iiro Rastas, Filip Ginter Abstract We apply a simple method to machine translate datasets with span-level annotation using the DeepL MT service and its ability to translate formatted documents. Using this method, we produce a Finnish version of the SQuAD2.0 question answering dataset and train QA retriever models on this new dataset. We evaluate the quality of the dataset and more generally the MT method through direct evaluation, indirect comparison to other similar datasets, a backtranslation experiment, as well as through the performance of downstream trained QA models. In all these evaluations, we find that the method of transfer is not only simple to use but produces consistently better translated data. Given its good performance on the SQuAD dataset, it is likely the method can be used to translate other similar span-annotated datasets for other tasks and languages as well. All code and data is available under an open license: data at HuggingFace TurkuNLP/squad_v2_fi, code on GitHub TurkuNLP/squad2-fi, and model at HuggingFace TurkuNLP/bert-base-finnish-cased-squad2. 1 Introduction Question answering (QA) is an important practical information retrieval task as well as a common benchmark of computational models of human language. Extractive QA models are typically built as a two step retriever-reader pipeline, first retrieving the documents relevant to the query (retriever) and then using an encoder model to extract the correct answer span from those documents (reader). Generative QA models replace the reader component with a generative large language model (LLM), in an approach commonly referred to as retrieval-augmented generation (RAG). No matter which of the QA paradigms is applied, large-scale question answering datasets such as the SQuAD dataset play a key role. Both in terms of benchmarking model performance, and model training. Whereas for extractive QA these datasets are used directly, in generative LLM development, QA datasets are commonly used as a source of examples for instruction fine-tuning. Unfortunately, these large-scale QA datasets are mostly available only for English and a small number of well-resourced languages, making the direct development of retriever-reader QA models for languages without such a dataset almost impossible, as well as negatively impacting benchmarking of LLM-based QA. With the improvements to machine translation (MT) output quality seen in the recent years, machine translating datasets is becoming a frequent choice to obtain a dataset in a new language in cases where native annotation is not possible due to lack of resources. While such an approach is technically very simple to implement for datasets consisting of unannotated text, it becomes considerably more complex for datasets with dense text span annotations, such as the QA datasets. Numerous approaches have been introduced aiming to transfer the span annotations during translation. In this paper we contribute to this overall line of research by demonstrating a simple, yet effective approach to translate English question answering datasets to Finnish (or other languages) using a little-known feature of the DeepL machine translation service. The primary contribution of this paper is a Finnish version of the publicly available sections of the SQuAD 2.0 dataset. This dataset can serve both for the development of extractive QA systems on top of Finnish encoder models, as well as provide a source of Finnish data for instruction tuning and benchmarking of Finnish LLMs. Our other contribution is a Finnish extractive QA model trained on this dataset. The paper is organized as follows: In Section 2 we review prior work on machine translating QA-datasets. In Section 3 we explain our process of machine translating these datasets. In Section 4 we evaluate the new resource and compare it to other similar resources. Finally, Section 5 concludes the work. 2 Related Work There are numerous open-domain question answering datasets for English. Among the most commonly used is the Stanford Question Answering Dataset (SQuAD). SQuAD1.1 Rajpurkar et al. 2016 consists of 100,000 questions posed by crowdsourced workers on a set of text passages (paragraphs of 536 Wikipedia articles). The questions are produced by the workers, while the answers constitute spans present in the text passages. SQuAD2.0 Rajpurkar et al. 2018 is a superset of SQuAD1.1 with an additional 50,000 crowdsourced unanswerable distractor questions that only make the impression of being answered in the given passage. 1 1 1 Note, however, that the test set of the SQuAD datasets is kept private, and the publicly available data contains 98,169 question-answer pairs for SQuAD1.1 and 92,749 answerable plus 49,434 unanswerable questions in SQuAD2.0. Native human generated question answering datasets for other languages include Chinese Cui et al. 2019 , Korean Lim et al. 2019 and French d’Hoffschmidt et al. 2020 , but a large number of languages lack a large QA dataset. The SQuAD dataset has been machine translated to several languages. Arabic Mozannar et al. 2019 SQuAD1.1 version starts by machine translating the passages, questions and answers separately. Subsequently, all the paragraphs and answers are transliterated to Arabic and the span of text of length at most 15 words with the least edit-distance with respect to the answer is identified. Only 231 articles containing 48,344 question-answer pairs are translated, and a full 25,490 question-answer pairs are not recovered by the initial translation and the transliteration heuristic step is applied. A reported small-scale evaluation shows that approximately 64% of these are correctly recovered. Persian Abadani et al. 2021 SQuAD2.0 version starts by machine translating the passages, questions and answers separately. Then an alignment is established by finding the position of the sentence that the answer appears in the English dataset. If the translated answer does not appear in the equivalent translated sentence, the question-answer pair is removed from the final dataset. The final dataset salvages 70,560 question-answer pairs. The TAR-method (Translate-Align-Retrieve) used to create the Spanish translation of SQuAD1.1 Carrino et al. 2019 also starts by machine translating the passages, questions and answers separately. If the translated answer can be found in the translated passage, it is retrieved as is. In the opposite case, a word alignment between the source and translated passage is established using the eflomal word alignment method Östling and Tiedemann 2016 and this alignment is then used to locate the translated answer. The final dataset salvages almost all of the question-answer pairs, but a manual error analysis showed that 50% of the answer spans were either misaligned (7%) or under-/over-extended (43%). For Finnish, which is our target language of interest, there exists an earlier machine translated version of the SQuAD2.0 dataset Kylliäinen 2022 ; Kylliäinen and Yangarber 2023 . The passages, questions and answers are translated separately and their spans in the translations are identified using a number of normalization steps designed to improve the chance of successful matching. The dataset preserves 66,000 question-answer pairs from the original approx. 92,000. The unpublished Swedish translation of SQuAD2.0 2 2 2 https://towardsdatascience.com/swedish-question-answering-with-bert-c856ccdcc337 deviates from the common approach, and translates one question-answer pair at a time, marking the answer span with a recognizable token (e.g. “[0]”), and retrieves the span after translation, relying on the MT system preserving the special tokens. This process is reported to preserve 90% of the original question-answer pairs. In a more recent approach, a separate alignment model is first trained for the target language Masad et al. 2023

Read the complete text