Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations
Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö · 2026
Read Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations by Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14815.txt.
Introduction
Luisto and seven others continued pretraining FinBERT, the Finnish BERT model, on unlabeled Finnish histopathology reports. Training-loss curves differed strongly by domain, a signal worth noting for practitioners. Since the reports carry no classification labels, the team tested whether pretraining signals track downstream gains using public Finnish datasets as proxies. In exploratory analysis, certain pretraining-derived features did correlate with proxy classification improvement. The report adds to the thin literature on natural language processing for Finnish healthcare data.
Catalog record
| CREATOR | Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33517 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.14815
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Domain Fine-Tuning FinBERT on Finnish Histopathological Reports: Train-Time Signals and Downstream Correlations Rami Luisto, Liisa Petäinen, Tommi Grönholm, Jan Böhm, Maarit Ahtiainen, Tomi Lilja, Ilkka Pölönen, Sami Äyrämö Abstract In NLP classification tasks where little labeled data exists, domain fine-tuning of transformer models on unlabeled data is an established approach. In this paper we have two aims. (1) We describe our observations from fine-tuning the Finnish BERT model on Finnish medical text data. (2) We report on our attempts to predict the benefit of domain-specific pre-training of Finnish BERT from observing the geometry of embedding changes due to domain fine-tuning. Our driving motivation is the common 2 2 2 First author’s professional anecdote. situation in healthcare AI where we might experience long delays in acquiring datasets, especially with respect to labels. 1 Introduction Ever since ULMFiT 13 the idea of separating the training of a language model to different stages of specialization (typically pre-training, domain fine-tuning and task-specific training) has been a core tenet in any modern NLP. This approach is particularly suitable for models based on the transformer architecture, see e.g. 39 ; 6 ; 20 ; 32 . Recent work has continued to study the specific benefits of domain fine-tuning, e.g. in the catchily titled Don’t stop pretraining: Adapt language models to domains and tasks 10 study how Domain-adaptive Pre-Training (DPT) compares to Task-Adaptive Pre-Training (TAPT) 1 1 1 In this paper we do not make a differentiation at this resolution, and just talk of Domain Fine-Tuning , DFT. . Especially in the realm of Large Language Models (LLMs) it can be crucial to predict from smaller exploratory training runs what would be the optimal configurations of model weights, train data amount and compute budget. The literature records various power laws that can be used to extrapolate model training behaviour. Such power laws exist both for comparing models within a data domain, see e.g. 15 ; 12 and the references within, or to compare behaviour across domains, see 4 . Though such work is typically focused on LLMs instead of Small Language Models (SLMs) like BERT, the underlying transformer architecture is the same, and many of the effects transfer. Figure 1 : Observations on the train-time loss of FinBERT on various datasets it is more or less “familiar” with. Histopathological dataset shows massive changes in loss. In the current work our aim is twofold. First we study how we might observe the effect of DFT on a dataset where we do not have training labels available. Second we try to replicate and observe the benefits that DFT have towards a classification task, with an eye on DFT training time observations that might correlate with classification task performance. We emphasize that our aim is to observe the phenomenon and make qualitative assessments. In particular, we do not aim for optimal classification algorithms as we wish to observe performance differentials rather than absolute levels. Our driving motivation here is NLP work in the healthcare sector, in particular with regards to Finnish medical text data. Finnish is a minority language with complex grammatical structures, meaning off-the-shelf multilingual models or translation-based approaches often fall short 26 . Furthermore, a specific feature of any AI work in the healthcare sector is that datasets are delicate and heavily regulated 2 ; 16 . It is not uncommon that in a healthcare project we might end up with only a partial dataset, e.g. only text without labels, either due to a mistaken data request or due to the fact that the labeling requires expert medical knowledge. The process of acquiring said labels can then take months of calendar time. We were interested to see here if we could use this wait time to do a DFT run on the data we have, and predict if it would be useful for the downstream task based on data we observe during the task. And so in this paper we present our results on running a DFT for the FinBERT model 40 on a medical histopathological dataset obtained from the Central Finland Biobank where we lacked the means to observe class labels for the data. We discuss what we were able to observe of the model during and after the training; the central observation is in Figure 1 . We then compare this to similar DFT runs against publicly available Finnish datasets. To base our observations, we also ran various classification tasks on these public datasets to see if there would be clear correlations between task performance and DFT run observations. We found a few tentative correlations, though a replication study will be needed. 1.1 Domain adaptation in low-resource medical languages While domain-adaptive pre-training has been extensively studied for English medical texts (e.g., BioBERT 19 , ClinicalBERT 1 ), applying these techniques to minority or low-resource languages can present special challenges. The relative scarcity of available healthcare data limits the tools available, and multilingual models are not always competitive with language-specific models, see 40 . Recent literature also highlights the necessity of localized, in-domain data for effective adaptation. For instance, 26 demonstrated that when extracting information from unstructured Finnish radiology reports, translating the reports to English to leverage existing models yielded worse results than using native Finnish texts. Similar observations have been made across other low-resource languages. In Turkish, 38 found that continuing pre-training on even a small, highly specific in-domain corpus was more effective for clinical text classification than utilizing broader, generic biomedical corpora. For European Portuguese, the MediAlbertina model 28 drastically reduced perplexity and improved information extraction simply by undergoing domain adaptation on local electronic medical records. Likewise, 5 showed that fine-tuning open-source LLMs on compiled Vietnamese medical textbook and forum data significantly improved health communication performance, bypassing the heavy “English bias” inherent in the base models. 2 Models, Data, Methods and Experiments Here we discuss the broad strokes of the technical aspect of our project. We start by going through the models we use, after which we discuss our various datasets in some detail. Finally we describe the measurements we did, both for analyzing the DFT effects and the classification results. 2.1 Models and their training Our work in this paper revolves around the Finnish language BERT variant TurkuNLP/bert-base-finnish-cased-v1 (which we’ll refer here as FinBERT ) by 40 . In many cases the more modern SLM variants like RoBERTa 20 or ModernBERT 41 would be more suitable for this kind of testing, but for the low-resource language of Finish a language-specific model is important for us – as noted in 40 , specialized language models can clearly outperform fine-tuned multilingual ones. Furthermore, for our qualitative observational analysis here the best possible performance is not an issue. From this core model we will train several fine-tuned versions on various datasets, and then use these for classification tasks. For DFT we use the Masked Language Model (MLM) task 2 2 2 Also known as the Cloze task in the more classical literature. where we have the model predicting randomly masked tokens with a cross-entropy loss. For the classification task we ran several kinds of variants, but for the sake of brevity we limit ourselves to discussing approaches where we trained a simple logistic regression or a k