DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing
Lisa Kluge, Maximilian Kähler · 2025
Read DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing by Lisa Kluge, Maximilian Kähler (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-21589.txt.
Introduction
Lisa Kluge and Maximilian Kähler describe the German National Library's system for automatically assigning subject keywords to catalog records with an ensemble of language models. The system entered a shared task on tagging a technical library's open-access catalog. It prompts several models with examples of human-indexed records, then asks them to suggest keywords for new ones.
Post-processing maps the free-text suggestions onto the official vocabulary. Votes across the models are combined and ranked by relevance to each record. The system placed fourth in the quantitative ranking of its track. In the qualitative judging by professional indexers, it came first. The result suggests machine suggestions are nearing professional quality for routine cataloging.
Catalog record
| CREATOR | Lisa Kluge, Maximilian Kähler |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-21589
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DNB-AI-Project at SemEval-2025 Task 5: An LLM-Ensemble Approach for Automated Subject Indexing Lisa Kluge, Maximilian Kähler Abstract This paper presents our system developed for the SemEval-2025 Task 5: LLMs4Subjects: LLM-based Automated Subject Tagging for a National Technical Library’s Open-Access Catalog. Our system relies on prompting a selection of LLMs with varying examples of intellectually annotated records and asking the LLMs to similarly suggest keywords for new records. This few-shot prompting technique is combined with a series of post-processing steps that map the generated keywords to the target vocabulary, aggregate the resulting subject terms to an ensemble vote and, finally, rank them as to their relevance to the record. Our system is fourth in the quantitative ranking in the all-subjects track, but achieves the best result in the qualitative ranking conducted by subject indexing experts. 1 Introduction The LLMs4Subject task D’Souza et al. 2025 aims at utilising large language models (LLMs) for the task of automated subject indexing on a dataset of open-access publications. Automated subject indexing is a task that helps enabling access to user-relevant publications by identifying and recording their most important themes and topics in the tagged subject terms. The ever-growing number especially of digital publications requires reliable automated systems for this task, which has become infeasible to achieve manually. In our previous work on automated subject indexing on a similar dataset Kluge and Kähler 2024 , we found that the performance of LLMs, while succesfully applied to a range of other tasks Zhao et al. 2023 ; Yang et al. 2024 ; Patil and Gudivada 2024 , was not yet on par with classical supervised machine learning methods. Therefore, it is important to do further research on the capabilities of LLMs in this context. Rather than fine-tuning models ourselves, the main strategy of our system is to leverage the existing capabilities of off-the-shelf foundational or instruction-tuned open-weight LLMs. In contrast to our previous work, the key contribution of this system is that it does not rely on only one LLM, but a combination of different language models along with varying prompts to generate the subject terms. We found this ensemble approach to dramatically improve the performance of our system. To handle the challenge of the controlled vocabulary unknown to the LLMs, we first generate free keywords with generative LLMs and then map these onto the vocabulary with a smaller embedding model. The official quantitative results put us in fourth place, the qualitative results even in first place. We think that our approach provides valuable insights into the chances and bounds of the few-shot prompting approach, showing that competitive results are possible without fine-tuning and large training corpora, simply by combining several LLMs into an ensemble. Our code is publicly available. 1 1 1 https://github.com/deutsche-nationalbibliothek/semeval25_llmensemble 2 Background Outlining the field of automated subject indexing, Golub 2021 presented important fundamentals, approaches and best practices for the task. Referring to it as index term assignment, Erbs et al. 2013 compared and combined two strategies to perform this task: multi-label classification (MLC) and keyword extraction. Detecting separate strengths, their results aligned with Toepfer and Seifert 2020 , who also found the combination of approaches to be beneficial. Regarding frameworks for automated subject indexing, the Annif system Suominen 2019 is an important contribution. Annif has established methods built in, like Omikuji 2 2 2 https://github.com/tomtung/omikuji , which is based on partitioned-label-tree-method Bonsai Khandagale et al. 2020 , or MLLM 3 3 3 https://github.com/NatLibFi/Annif/wiki/Backend:-MLLM , a lexical approach building on Medelyan 2009 ’s Maui. In earlier work Kluge and Kähler 2024 , we presented experiments with a closed-source LLM on automated subject indexing, but the two baseline methods implemented in Annif mentioned above were found to be as good as or even outperform our LLM-based method. LLMs have also been utilised for MLC Peskine et al. 2023 ; D’Oosterlinck et al. 2024 ; Zhu and Zamani 2024 and keyword extraction Maragheh et al. 2023 ; Lee et al. 2023 . Recently, building ensembles or fusioning (the results of) LLMs has been addressed as a promising research direction. There are different works sharing the idea of exploiting the individual strengths and diminishing the weaknesses in different LLMs Jiang et al. 2023b ; Lu et al. 2023 ; Wang et al. 2023 ; Fang et al. 2024 ; Wan et al. 2024 . Exploring the goal of building ensembles, Tekin et al. 2024 aimed at maximising diversity and efficiency, whereas Chen et al. 2023 targeted the reduction of inference cost. Not only LLMs have been combined, but also prompts Pitis et al. 2023 ; Hou et al. 2023 . Combining both prompts and models on the task of phishing detection, Trad and Chehab 2024 contrasted prompt-based ensembles (with one prompt and several LLMs), model-based ensembles and an ensemble consisting of a mixture of prompts and models. 3 System Overview Our system is an enhancement from our previous LLM-based subject indexing approach, described in Kluge and Kähler 2024 . In total, it consists of 5 stages, complete , map , summarise , rank and combine , as depicted in the overview in Figure 1 . At its core, the system approaches the subject indexing task as a keyword generation problem which is solved by a few-shot prompting LLM procedure. As these generated keywords are a priori not restricted to the target vocabulary, a mapping stage with a smaller word embedding model is needed as a supplementary step. In comparison to our previous approach, we have extended the system by combining multiple LLMs and prompts to an ensemble and by introducing an LLM-powered ranking step as in D’Oosterlinck et al. 2024 . Figure 1: Illustration of our LLM-ensemble approach. 3.1 Complete The first step in our subject indexing system, complete , is the generation of keywords following the few-shot paradigm, similar to the procedure in Lee et al. 2023 . The complete -step is repeated over a range of diverse off-the-shelf open-weight LLMs and prompts with varying few-shot examples. Our plan and intention of employing a broad variety of models and prompts are twofold. In comparison to a single-model single-prompt setting, we aim to: • Improve recall with an overall greater set of generated subject terms in the ensemble. • Improve precision by utilising the overlap of various model \times prompt combinations. Each prompt consists of an instruction and a set of 8-12 examples illustrating how to perform the subject indexing task with example texts and their gold-standard subject terms. Details for LLM selection and the composition of the few-shot prompts will be discussed in Sections 4.3 and 4.4 . 3.2 Map Keywords generated in the first stage are mapped to controlled subject terms in the target vocabulary using a word embedding model as in Zhu and Zamani 2024 . For our map -stage, we used Chen et al. 2024 ’s BGE-M3-embeddings. Both generated keywords and target vocabulary are embedded with the same model. 4 4 4 We embedded the keywords and vocabulary entries without integrating them into a context sentence (which we did in our previous approach). To perform nearest neighbour search, we uploaded the embeddings to a Weaviate 5 5 5 https://weaviate.io/ vector storage enabling efficient HNSW-Search Malkov and Yashunin 2016 in \mathcal{O}(\log L) complexity,