FuDoBa: Fusing Document and Knowledge Graph based Representations with Bayesian Optimisation
Boshko Koloski, Senja Pollak, Roberto Navigli, Blaž Škrlj · 2025
Read FuDoBa: Fusing Document and Knowledge Graph based Representations with Bayesian Optimisation by Boshko Koloski, Senja Pollak, Roberto Navigli, Blaž Škrlj (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-06622.txt.
Introduction
Boshko Koloski and co-authors present FuDoBa, a method that fuses two views of a document, in this 2025 paper. One view comes from a large language model; the other from structured knowledge graphs, local or public ones such as Wikidata. Model embeddings alone can be huge, costly, and too generic for narrow fields.
FuDoBa uses Bayesian optimization to find a good blend, yielding small, task-fit representations. It also reports the blend weights, which keeps the result readable. On six datasets in two domains, the fused views, paired with automated classifiers, matched or beat embeddings from big proprietary models.
Catalog record
| CREATOR | Boshko Koloski, Senja Pollak, Roberto Navigli, Blaž Škrlj |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 61 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-06622
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FuDoBa: Fusing Document and Knowledge Graph based Representations with Bayesian Optimisation Boshko Koloski, Senja Pollak, Roberto Navigli, Blaž Škrlj Abstract Building on the success of Large Language Models (LLMs), LLM-based representations have dominated the document representation landscape, achieving great performance on the document embedding benchmarks. However, the high-dimensional, computationally expensive embeddings from LLMs tend to be either too generic or inefficient for domain-specific applications. To address these limitations, we introduce FuDoBa—a Bayesian optimisation-based method that integrates LLM-based embeddings with domain-specific structured knowledge, sourced both locally and from external repositories like WikiData. This fusion produces low-dimensional, task-relevant representations while reducing training complexity and yielding interpretable early-fusion weights for enhanced classification performance. We demonstrate the effectiveness of our approach on six datasets in two domains, showing that when paired with robust AutoML-based classifiers, our proposed representation learning approach performs on par with, or surpasses, those produced solely by the proprietary LLM-based embedding baselines. keywords document classification, Bayesian optimisation, representation learning, knowledge graphs, early fusion, multimodal learning 1 Introduction Efficient and rich document representations are the building blocks for many natural language processing (NLP) tasks such as classification or clustering Muennighoff et al. 2023 . Contemporary methods for representing documents focus on distilling representations from either pre-trained language models (PLMs) such as BERT Devlin et al. 2019 or large language models (LLMs) such as Llama3 Grattafiori and et al. 2024 , exploiting the rich semantic knowledge acquired during pre-training on vast text corpora. For instance, Sentence-BERT Reimers and Gurevych 2019 builds document representation by pooling over pre-trained BERT-based word embeddings, which are further refined through contrastive learning and Siamese networks. Similarly, LLM2Vec BehnamGhader et al. 2024 disentangles the causal masking of LLMs to a bi-directional one, further post-training the LLM on a masked next token prediction task and finally, training with a contrastive training objective, similarly to Sentence-BERT, refining the final representations via mean pooling by training with a contrastive training objective. Despite good performance on public benchmarks such as MTEB Muennighoff et al. 2023 , contrastive pre-training models require acquiring a dataset of triplet sentences (i.e., query, positive answer, and negative answer), which is often infeasible and costly. That said, even assuming such a dataset is available, training is costly and requires extra compute. The best-performing representations to date often come from proprietary companies and their mechanisms are largely unknown Muennighoff et al. 2023 . Current approaches face two notable limitations: first, these embeddings typically reside in high-dimensional spaces, often exceeding 1,000 dimensions, creating practical challenges for efficient storage and computation (especially for AutoML model training); second, their generalist nature makes them suboptimal for domain-specific applications without expensive fine-tuning, which in some cases is infeasible. One way to overcome the costly training requirements while still leveraging the expressive power of PLM/LLM derived representations is to introduce additional knowledge from auxiliary representations. Previous studies proposed representation enrichment via additional representations, either via representation alignment Chen et al. 2022 or representation fusion Koloski et al. 2022 . In many cases, the approach of introducing auxiliary knowledge to document representations and searching over a space of possible classifiers with AutoML performs on par with task-specific fine-tuned PLMs such as BERT, without training costly classifiers Škrlj et al. 2021 . Recently, Koloski et al. 2025 proposed BabelFusion, involving the introduction of global knowledge from a knowledge graph by entity-linking to BabelNet’s Navigli and Ponzetto 2010 synsets connected to a subgraph of WikiData5m Wang et al. 2021 . They showed that projecting the inputs into low-dimensions not only produces low-dimensional spaces but can generate models that usually out-perform the language-only representations in the high dimensional space. However, they concluded that matching datasets where text stems from online discussion and is usually manifested with short texts with against a knowledge graph was sub-optimal. Indeed, this proved deteriorating for the downstream performance of AutoML classifiers, in comparison to that of weak classifiers fitted on the original space. As an alternative to their approach, we hypothesise that extracting domain-specific knowledge graphs, resulting in a local knowledge graph 1 1 1 We refer to them as LocKG throughout the remainder of the paper. , can provide local context that could complement to a global knowledge graph such as WikiDAta, aiding LLMs downstream. In this work, we propose FuDoBa (Figure 1 ), a novel Bayesian optimisation-based early fusion methodology that builds on the idea of Koloski et al. 2025 . It combines the semantic richness of LLM representations with the structured information from knowledge graphs and proposes the construction of local knowledge by extracting structured relations (e.g., knowledge triplets) from the dataset using Relation Extraction methods. Our approach systematically leverages external knowledge sources such as WikiData alongside these domain-specific, locally extracted knowledge structures to create more contextualised and task-relevant document representations. By employing Bayesian optimisation techniques, we identify importance parameters that maximize performance while minimising dimensionality. Furthermore, the optimised global weights assigned by Bayesian optimisation to the different representation sources serve as a a task-specific interpretation of the embeddings’ importance. The main contributions of this work are as follows: • We demonstrate that LLM-based embeddings can be further improved by integrating both global contextual information and fine-grained, domain-specific local knowledge. In particular, we systematically evaluate the impact of constructing and incorporating a domain-specific knowledge graph—capturing domain dependant relations between entities. • We experimentally demonstrate that training AutoML in low-dimensional spaces yields downstream models that perform as well as—or even better than—those produced using conventional methods. • We introduce a novel, Bayesian optimisation–based method for low-dimensional early fusion of document and knowledge graph representations. The remainder of this paper is structured as follows. Section 2 reviews the relevant literature, and Section 3 outlines the proposed methodology. Section 4 describes the experimental setup, and Section 5 presents the results. Section 6 discusses the findings, and Section 7 concludes the paper. The Appendix details the method’s limitations in Section 8 , implementation specifics in Section 9 , a deeper analysis of relation extractors within the local knowledge graph in Section 10 , and additional experimental results in Section 11 . Figure 1: Overview of our proposed FuDoBa framework for representation enrichment. Starting from an LLM-based embedding (purple) external knowledge is incorporated via two parallel pathways: a global knowledge graph (light blue) branch that links entities using Babelfy Moro et al. 2014 and WikiData5m Wang et al. 2021 embeddings, and a novel local knowledge graph branch (yellow) that constructs domain-specific knowledge-graph via relation-extraction models. Representations are projected into a lower-dimensional space ( l ) and weighted by an importance parameter ( \alpha ), with ElasticNet normalisation applied both before and after fusion. The concatenated representation is then processed by AutoGluon Erickson et al. 2020 AutoML for model search under a constrained time budget. 2 Related Work The task of document representation has evolved from pre-training shallow neural networks (NNs) on web scale data Le and Mikolov 2014 through transformer encoders Devlin et al. 2019 refined by contrastive learning Reimers and Gurevych 2019 ; Gao et al. 2021 ; Li and Li 2024 , to adaptations of decoder-only LLMs Aghajanyan et al. 2024 ; Khosla et al. 2025 and synthetic data approaches Wang et al. 2024 . While these techniques output powerful high-performant embeddings Muennighoff et al.