Enhancing BERTopic with Intermediate Layer Representations

Dominik Koterwa, Maciej Świtała · 2025

Read Enhancing BERTopic with Intermediate Layer Representations by Dominik Koterwa, Maciej Świtała (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06696.txt.

Introduction

Topic modeling sorts a pile of documents into themes, and BERTopic does it by clustering transformer embeddings of the texts. The result depends on how those embeddings are prepared, a step users rarely question. Dominik Koterwa and Maciej Świtała test 18 ways of building the embeddings, including pulling representations from intermediate layers of the model and transforming them. They run the comparison on three diverse datasets and score each setup for topic coherence and topic diversity.

For every dataset, some configuration beats the default BERTopic setting. The authors also study how stop words influence the different embedding choices. Their message is practical: the default is a starting point, not the best choice. Tuning the embedding layer can yield clearer, more varied topics from the same documents and the same core algorithm.

Catalog record

CREATORDominik Koterwa, Maciej Świtała
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06696

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Enhancing BERTopic with Intermediate Layer Representations Dominik Koterwa, Maciej Świtała Abstract BERTopic is a topic modeling algorithm that leverages Transformer-based embeddings to create dense clusters, enabling the estimation of topic structures and the extraction of valuable insights from a corpus of documents. This approach allows users to efficiently process large-scale text data and gain meaningful insights into its structure. While BERTopic is a powerful tool, embedding preparation can vary, including extracting representations from intermediate model layers and applying transformations to these embeddings. In this study, we evaluate 18 different embedding representations and present findings based on experiments conducted on three diverse datasets. To assess the algorithm’s performance, we report topic coherence and topic diversity metrics across all experiments. Our results demonstrate that, for each dataset, it is possible to find an embedding configuration that performs better than the default setting of BERTopic. Additionally, we investigate the influence of stop words on different embedding configurations. 1 Introduction Topic modeling (TM) is a popular tool in text mining which attempts to process multiple unlabeled documents and provide valuable insights about their interior, mainly by creating a representation of topics present in them. Throughout the years many topic modeling algorithms have been developed and applied in various fields, such as finance Aziz et al. (2022) ; Ogunleye et al. (2023) ; Raju et al. (2022) , social media Falkenberg et al. (2022) ; Li et al. (2024) ; Ramondt et al. (2022) , or medicine Jeon et al. (2023) ; Yao et al. (2018) ; Guizzardi et al. (2023) . Topic modeling has proven to be a useful tool to retrieve valuable insights about data from different modalities, for example images Feng and Lapata (2010) , video Hospedales et al. (2012) , and audio Gong and Poellabauer (2017) . Ongoing advancements are being pursued in the field of topic modeling, with novel methodologies and insights emerging progressively over time. One of the more recent advancements in topic modeling is BERTopic Grootendorst (2022) , a technique that leverages embeddings from Sentence Transformers Reimers and Gurevych (2019) and clustering methods to generate coherent and interpretable topics. Sentence Transformers typically consist of multiple stacked encoder layers that process information sequentially. Each encoder layer generates intermediate representations known as hidden states . Unlike traditional approaches such as Latent Dirichlet Allocation (LDA) Blei et al. (2003) or Non-negative Matrix Factorization (NMF) Paatero and Tapper (1994) , BERTopic incorporates contextualized word representations, allowing it to capture semantic relationships more effectively. This makes it particularly useful for analyzing large-scale text data across various domains. However, despite its growing popularity, the impact of different embedding representations for document encoding has not been yet studied. Most implementations rely on the default setting (Mean pooling with the last layer’s hidden states) without exploring alternative strategies that could potentially enhance quality. 2 Related work This research is mainly inspired by studies examining the geometry and structure of contextualized word representations. Several works have analyzed the influence of embeddings from different layers of Transformer-based models Vaswani et al. (2017) . Ethayarajh (2019) investigated the contextualization of word representations and the variations in encoding the same words in different contexts. Ma et al. (2019) evaluated embeddings extracted from various encoder layers on a range of downstream and probing tasks, demonstrating that upper-layer embeddings generally yield better performance. However, for certain probing tasks, such as tense classification as well as subject and object number classification, representations from middle layers achieve the best results. The study also examined the effect of pooling methods on model performance, showing significant differences in metric values between various pooling strategies. Jawahar et al. (2019) visualized how BERT embeddings shift across layers and conducted multiple probing experiments, concluding that different layers vary in their ability to encode syntactic and semantic information. Similarly, Puccetti et al. (2021) applied logistic regression Tibshirani (1996) to BERT activations and presented results on 68 sentence-level probing tasks. Their findings suggest that specific groups of hidden units are more relevant to particular linguistic properties and similar tasks. The influence of pooling strategies was also evaluated. Raganato and Tiedemann (2018) explored Transformer representations in a multilingual setting, revealing variations in probing task performance across different layers for all languages examined. Inspired by previous research, this study investigates whether embeddings extracted from different model layers significantly affect evaluation metrics in a topic modeling setting and whether alternative strategies lead to improved results. To explore these questions, we adopt the same architectural framework as in BERTopic paper. 3 Data We have decided to use exactly the same datasets evaluated by Grootendorst (2022) to make the results comparable between two works. Dataset # Documents Vocab Size Avg Words Most Common Words 20 Newsgroups 16,309 1,612 48.2 make, people, time good, work, year system, file, find give Trump Tweets 44,252 20,319 15.86 great, trump, thank people, would, get new, president, like big United Nations 50,422 29,157 68.38 united, security, nations development, world, must international, also, peace countries Table 1: Descriptive statistics of datasets used in experiments. The BBC News dataset Greene and Cunningham (2006) was replaced with transcriptions of United Nations (UN) General Debates from 2006 to 2015 1 1 1 https://runestone.academy/runestone/books/published/httlads/_static/un-general-debates.csv . The UN debates corpus was previously used in the BERTopic article but specifically in the dynamic topic modeling setting. This substitution was made due to the limited size of the BBC News dataset, which contained only 2,225 observations. During evaluation, the BERTopic algorithm reduces the number of topics from an initially calculated value to a user-specified parameter. For this dataset, the evaluation tool consistently detected fewer topics than the user-defined parameter. As a result, the calculated results showed minimal variation in nearly all tested configurations. Trump’s tweets 2 2 2 https://www.thetrumparchive.com/faq from 2009 to 2021, totaling 44,252 observations (excluding retweets), serve as a source of short-text data. Since BERTopic has been applied to tweets from various domains, analyzing this dataset can provide valuable insights across multiple fields. The 20 Newsgroups dataset 3 3 3 https://github.com/MIND-Lab/OCTIS/tree/master/preprocessed_datasets/20NewsGroup consists of 16,309 news articles across 20 categories. This dataset complements the other two, allowing for evaluations across three distinct domains. More details on the structure of the datasets are provided in Table 1 . (a) Possible to retrieve embeddings from the model. (b) Configurations of embeddings used in experiments. Figure 1: Visualization of embedding configurations used in our experiments, "Sum" refers to summing the embeddings, while "Concat" indicates concatenating the representations. 4 Experiments Our embedding configurations closely follow those used by Devlin et al. (2019) , with six distinct retrieval methods implemented. Embedding retrieval is performed by extracting hidden states from specific encoders. We then apply functions such as summation or concatenation to construct final embeddings for evaluation. The following representations are used: 1. Output from the last layer 2. Output from the embedding layer 3. Sum of outputs from all layers 4. Output from the second-to-last layer 5. Sum of outputs from the last four layers 6. Concatenation of outputs from the last four layers An illustration of the model’s possible outputs and the evaluated configurations is provided in Figure 1 . In addition, three pooling methods—Mean, Max, and CLS pooling—were incorporated into the experimental setup, resulting in a total of 18 configurations. To

Read the complete text