Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs

Joonatan Laato, Jenna Kanerva, John Loehr, Virpi Lummaa, Filip Ginter · 2025

Read Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs by Joonatan Laato, Jenna Kanerva, John Loehr, Virpi Lummaa, Filip Ginter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2502-13566.txt.

Introduction

Joonatan Laato and four co-authors mine old refugee interviews for social ties with language models in a 2025 paper. The archive holds some eighty nine thousand short Finnish interviews with families moved from Karelia after the Second World War. Clubs and hobbies named in the text can proxy how well the families wove into their new homes.

The team pulled groups and hobbies per family member with no task training. The best closed model matched human reads at near eighty nine percent on the joint score. The best open model landed just behind it, a strong sign for free tools on non English text. A Finnish tuned model fed on machine made training data scored a few points lower, but suits huge jobs on tight hardware.

Catalog record

CREATORJoonatan Laato, Jenna Kanerva, John Loehr, Virpi Lummaa, Filip Ginter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2502-13566

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extracting Social Connections from Finnish Karelian Refugee Interviews Using LLMs Joonatan Laato, Jenna Kanerva, John Loehr, Virpi Lummaa, Filip Ginter Abstract We performed a zero-shot information extraction study on a historical collection of 89,339 brief Finnish-language interviews of refugee families relocated post-WWII from Finnish Eastern Karelia. Our research objective is two-fold. First, we aim to extract social organizations and hobbies from the free text of the interviews, separately for each family member. These can act as a proxy variable indicating the degree of social integration of refugees in their new environment. Second, we aim to evaluate several alternative ways to approach this task, comparing a number of generative models and a supervised learning approach, to gain a broader insight into the relative merits of these different approaches and their applicability in similar studies. We find that the best generative model (GPT-4) is roughly on par with human performance, at an F-score of 88.8%. Interestingly, the best open generative model (Llama-3-70B-Instruct) reaches almost the same performance, at 87.7% F-score, demonstrating that open models are becoming a viable alternative for some practical tasks even on non-English data. Additionally, we test a supervised learning alternative, where we fine-tune a Finnish BERT model (FinBERT) using GPT-4 generated training data. By this method, we achieved an F-score of 84.1% already with 6K interviews up to an F-score of 86.3% with 30k interviews. Such an approach would be particularly appealing in cases where the computational resources are limited, or there is a substantial mass of data to process. [ [email protected], ] *1 [ [email protected], orcid=0000-0003-4580-5366, ] *1 [ [email protected], orcid=0000-0002-6212-0273, ] [ [email protected], orcid=0000-0002-2128-7587, ] [ [email protected], orcid=0000-0002-5484-6103, ] keywords information extraction ,LLM ,Karelian refugees ,zero-shot extraction ,Finnish † † copyrightyear: 2024 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: CHR 2024: Computational Humanities Research Conference, December 4–6, 2024, Aarhus, Denmark † † address: TurkuNLP, Department of Computing, University of Turku, Finland † † address: Lammi Biological Station, Faculty of Biological and Environmental Sciences, University of Helsinki, Finland † † address: Department of Biology, University of Turku, Finland † † corresp: Equal contribution 1 Introduction Numerous research questions in digital humanities require, in one form or another, extraction of factual information from very large textual collections ( Ehrmann et al. 2023 ) , often taking into account context spanning beyond the immediate textual neighbourhood. Traditionally, such tasks have been approached by first annotating a sufficiently large set of training instances, and subsequently building a dedicated supervised machine learning model for each task separately. Typically, these would be document- or entity-level annotations, such as document labels, named entity categories, and pairwise typed relations between the extracted entities. Recently, the zero-shot capabilities of large language models (LLMs) combined with decreasing inference costs, offer an attractive alternative path, whereby the same outcome is achieved by prompting a suitable generative LLM, with only a small number of instances informing the model of the task (referred to as zero-shot or in-context learning). However, it is still very much an open question, which tasks can be approached in this manner, what levels of accuracy can be expected, which models are suitable, and what limitations one should be aware of when applying such methodology. In this paper, we aim to add to this understanding, pursuing a case study on a large collection of short, Finnish-language family history interviews of post-WWII refugee families from the Finnish Karelia region. We address a context-dependent information extraction task on this dataset, intended to facilitate history research on the degree of social integration of refugee families in their new environment. In this paper, we nevertheless focus on the methodological rather than historical aspects of the work, aiming to answer the following: 1. Can LLM prompting process historical non-English data at a useful level of accuracy? 2. Are there substantial differences in the performance of various LLMs for the task? 3. Is it feasible to use LLMs to produce training data for a dedicated lightweight model? We evaluate OpenAI’s proprietary models, GPT-3.5 Brown et al. 2020 and GPT-4 Achiam et al. 2023 , and contrast their performance with various open-source models. Additionally, we compare the LLM performance to a much smaller and lightweight tagger model fine-tuned for the task using GPT-4 generated training data. Such an approach becomes particularly appealing in cases where running a generative language model for the full dataset would not be economical. The models are evaluated on a manually annotated sample, which we also use to estimate human performance on the task. 2 Related Work The recent conversational LLMs have been applied to various NLP tasks in a zero-shot setting, where the task instructions are given directly in the model prompt. While many such studies report encouraging performance (e.g. reasoning and dialogue ( Qin et al. 2023 ; Abaskohi et al. 2024 ) , text classification ( Törnberg 2023 ; Gilardi et al. 2023 ; Debess et al. 2024 ) , and relation classification ( Aguda et al. 2024 ) ), also negative results have been reported ( Edwards and Camacho-Collados 2024 ; Koneru et al. 2024 ) , especially when compared against task-specific supervised models. For example, in the area of named entity recognition (NER) and relation extraction (RE), both Wei et al. 2024 and Qin et al. 2023 report the generative LLMs substantially lagging behind the state-of-the-art supervised models in both tasks when evaluated on English. Abaskohi et al. 2024 report similar results when applying ChatGPT to Persian named entity recognition. While their NER results clearly lag behind supervised models, their experiments show the LLMs being highly competitive in some Persian tasks, indicating the model works well also on a language other than English. This is also supported by Debess et al. 2024 , who report GPT-4 performing remarkably well when compared to human annotators on Faroese sentiment analysis, as well as Tarkka et al. 2024 on Finnish emotion classification. Given the previous work, our study focuses on a task close to named entity recognition, however, in contrast to works by Wei et al. 2024 ; Qin et al. 2023 ; Abaskohi et al. 2024 , we do not attempt to create a universal method capable of returning any given named entities from any given input text. Rather we focus only on two types of entities (namely social organizations and hobbies) from a specific text collection (Karelian refugee interviews). Therefore, we can design the prompt specifically for the targeted entity types and text collection. Previously, a targeted information retrieval study has been conducted e.g. in Polak and Morgan 2023 , where a target-specific, multi-step pipeline was developed for extracting materials properties from English materials science articles, reaching close to 90% F-score using GPT-4. However, the text domain (English scientific articles compared to Finnish historical interviews), as well as targeted entity types (materials properties compared to persons’ social organizations and hobbies), greatly differ in our study. 3 Data Figure 1: Two interviews translated into English for illustration purposes. The relevant information (hobbies, social organizations) related to the primary person (the one being interviewed) is illustrated in yellow, while blue indicates the spouse. Our primary source of data

Read the complete text