Annif at the GermEval-2025 LLMs4Subjects Task: Traditional XMTC Augmented by Efficient LLMs

Osma Suominen, Juho Inkinen, Mona Lehtinen · 2025

Read Annif at the GermEval-2025 LLMs4Subjects Task: Traditional XMTC Augmented by Efficient LLMs by Osma Suominen, Juho Inkinen, Mona Lehtinen (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15877.txt.

Introduction

Osma Suominen, Juho Inkinen, and Mona Lehtinen report a shared task entry in a 2025 paper. The task asked systems to assign subject labels to library records. Efficiency mattered as much as accuracy in the scoring.

Their Annif toolkit starts from traditional automated indexing methods. Small language models translate records and create synthetic training data. Larger model calls are reserved for ranking candidate subjects. That division keeps the computing bill low by design. The system placed first in both quantitative and qualitative judging. Readers see frugal AI beat heavier rivals in a library task.

Catalog record

CREATOROsma Suominen, Juho Inkinen, Mona Lehtinen
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-15877

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Annif at the GermEval-2025 LLMs4Subjects Task: Traditional XMTC Augmented by Efficient LLMs Osma Suominen, Juho Inkinen, Mona Lehtinen Abstract This paper presents the Annif system in the LLMs4Subjects shared task (Subtask 2) at GermEval-2025. The task required creating subject predictions for bibliographic records using large language models, with a special focus on computational efficiency. Our system, based on the Annif automated subject indexing toolkit, refines our previous system from the first LLMs4Subjects shared task, which produced excellent results. We further improved the system by using many small and efficient language models for translation and synthetic data generation and by using LLMs for ranking candidate subjects. Our system ranked 1st in the overall quantitative evaluation of and 1st in the qualitative evaluation of Subtask 2. 1 Introduction Subject indexing is an important aspect of improving the discoverability of bibliographic databases and digital collections. The LLMs4Subjects Shared Task at GermEval-2025 invited teams to produce innovative systems for the application of large language models (LLMs) to the task of automated subject indexing. We participated in the shared task D’Souza et al. 2025 with an Annif 1 1 1 https://annif.org -based system, as we have been developing the Annif automated subject indexing toolkit since 2017 Suominen et al. 2022 . Our system is a more refined version of the one we used in the first LLMs4Subjects shared task at SemEval-2025 Suominen et al. 2025 . The novel aspects of our work are 1) translation of metadata records using many different LLMs and choosing an optimal compromise between throughput and quality, 2) generation of synthetic training data using different LLMs in combination, 3) a fusion approach involving optimised weights and exponents, and 4) using a LLM for ranking candidate subjects in an ensemble. Our system ranked first within Subtask 2 in both quantitative and qualitative evaluations. We demonstrated that our approach that combines LLMs with traditional natural language processing and machine learning remains competitive against other systems that use LLMs more heavily, and allows for different compromises between efficiency and quality - finding the best ones is important in and as of itself. Our code, configuration files and customised data sets are available on GitHub 2 2 2 https://github.com/NatLibFi/Annif-LLMs4Subjects-GermEval2025/ . The models we trained are available on Hugging Face Hub 3 3 3 https://huggingface.co/NatLibFi/Annif-LLMs4Subjects-GermEval2025-data . 2 Background We only participated in Subtask 2: Subject Indexing. The task involved developing LLM-based systems that recommend the most relevant subjects from the GND subject vocabulary (200,035 subjects) to tag a given TIBKAT record based on its title and abstract. This is an extreme multi-label text classification (XMTC) problem. The organisers provided bibliographic records from the TIBKAT database as a data set D’Souza et al. 2025 divided into train (n=90452), development (n=19949), and test (n=27998) subsets. Subsets were further split by document type (e.g. Article or Book) and language (German or English). GND subjects were included for the train and development records. As in our previous work, our system relied mainly on the traditional XMTC algorithms in Annif. However, we used LLMs to pre-process the data sets, to generate additional synthetic training data, and for the final ranking of candidate subjects. 3 System overview As in our previous work, we based our system on the Annif automated subject indexing toolkit that provides a selection of XMTC algorithms. We set up base projects using three Annif backends: 1) Omikuji 4 4 4 https://github.com/tomtung/omikuji , an implementation of a family of efficient machine learning algorithms for multilabel classification based on the idea of partitioned label trees, including Parabel Prabhu et al. 2018 and Bonsai Khandagale et al. 2020 . We used the Bonsai-style configuration. 2) MLLM 5 5 5 https://github.com/NatLibFi/Annif/wiki/Backend%3A-MLLM (Maui-like Lexical Matching), a lexical algorithm for matching words and expressions in document text to terms in a subject vocabulary. It is a reimplementation of the ideas behind Maui Medelyan 2009 . 3) XTransformer , an XMTC and ranking algorithm based on fine-tuned BERT-style Transformer models that is part of the PECOS framework Yu et al. 2022 . We used FacebookAI/xlm-roberta-base as the base model. ID HuggingFace ID Size A8 CohereLabs/aya-expanse-8b 8 E9 utter-project/EuroLLM-9B-Instruct 9 G4 google/gemma-3-4b-it 4 G12 google/gemma-3-12b-it 12 H3 NousResearch/Hermes-3-Llama-3.2-3B 3 H8 NousResearch/Hermes-3-Llama-3.1-8B 8 I8 ibm-granite/granite-3.3-8b-instruct 8 L3 meta-llama/Llama-3.2-3B-Instruct 3 L8 meta-llama/Llama-3.1-8B-Instruct 8 M8 mistralai/Ministral-8B-Instruct-2410 8 M12 mistralai/Mistral-Nemo-Instruct-2407 12 Q4 Qwen/Qwen3-4B 4 Q8 Qwen/Qwen3-8B 8 S8 VAGOsolutions/Llama-3-SauerkrautLM-8b-Instruct 8 S12 VAGOsolutions/SauerkrautLM-Nemo-12b-Instruct 12 G27 google/gemma-3-27b-it 27 M24 mistralai/Mistral-Small-3.1-24B-Instruct-2503 24 Q30 Qwen/Qwen3-30B-A3B 30 Table 1: LLMs used in our system. Since a special theme for this shared task was energy- and compute-efficient LLMs , we specifically looked at opportunities to reduce computational costs while maintaining high performance. Thus, we mainly applied smaller, locally run (see Appendix C ) language models and evaluated their relative performance. We selected recently published open-weight LLMs, ranging in size from 3B to 12B parameters, with support for both English and German language, including models specifically fine-tuned for improved support of German (SauerkrautLM) as well as general purpose fine-tunes (Hermes 3). We also included three models in a larger size category (24B to 30B) for comparison. The language models we used are shown in Table 1 ; we use the mnemonic model IDs listed in the table in the remainder of the paper. We combined the base backends into two types of ensembles: simple ensembles that merge subject suggestions from two or more backends by averaging their scores and a new LLM ranking ensemble . The simple ensemble was improved from our previous work by adding support for weighting exponents (see Appendix A for details). The LLM ranking ensemble, in addition to averaging scores, also uses an external LLM to rank candidate subjects similar to the DNB-AI-Project system Kluge and Kähler 2025 in the first LLMs4Subjects Shared Task (see Appendix B for details). For evaluation during system development, we used common XMTC metrics built in to the Annif toolkit: F1@5 (F1 score calculated using the top 5 suggestions from the system) and nDCG@20 Järvelin and Kekäläinen 2002 , a ranking metric calculated using the top 20 suggestions from the system. We used the Data Version Control 6 6 6 https://dvc.org/ tool to manage the data sets, project configurations and the training and evaluation processes. 4 Experimental setup Figure 1: LLM pre-processing steps for the data sets. 4.1 Translation of data sets For the vocabulary, we reused the bilingual English-German SKOS version of GND that was produced in our earlier work by translating the preferred terms into English using the GPT-4o-mini LLM. To translate the titles and abstracts of the TIBKAT records into a consistent target language, we tested many different LLMs, aiming to find a good balance between computational efficiency and subject indexing quality. We translated each record from the train and development sets separately into German-only and English-only variants (step 1 in Figure 1 ). We then trained Omikuji Bonsai models on the translated train records and evaluated the results against the corresponding development set. We scored the LLMs separately for English and German target languages using the formula score=nDCG@20+\alpha\cdot throughput , which combines quality and efficiency into a single metric, selecting \alpha=0.003 as the balancing factor. The value for the factor \alpha was chosen based on experimentation and the results are shown in Figure 3

Read the complete text