Topic Segmentation Using Generative Language Models

Pierre Mackenzie ⋆,† , Maya Shah ⋆,‡ , Patrick Frenett ⋆ , ⋆ Adarga, † University of Edinburgh, ‡ University of Exeter, Correspondence: lardet[dot]pierre[at]gmail.com · 2026

Read Topic Segmentation Using Generative Language Models by Pierre Mackenzie ⋆,† , Maya Shah ⋆,‡ , Patrick Frenett ⋆ , ⋆ Adarga, † University of Edinburgh, ‡ University of Exeter, Correspondence: lardet[dot]pierre[at]gmail.com (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-03276.txt.

Introduction

Pierre Mackenzie, Maya Shah, and Patrick Frenett test whether generative language models can split documents into topical segments. Earlier segmenters mostly compare sentence meanings, which misses the long-range dependencies and broad knowledge that large models carry. The authors propose an overlapping, recursive prompting strategy built on sentence enumeration, feeding the model numbered sentences and asking where topics change.

They also support adopting the boundary similarity metric for scoring segmentation quality. Their results show language models can segment more effectively than existing methods. Still, the authors are clear that open issues remain before such models can be relied on for topic segmentation in practice. The study maps both the promise and the current limits of prompting for this classic text-processing task.

Catalog record

CREATORPierre Mackenzie ⋆,† , Maya Shah ⋆,‡ , Patrick Frenett ⋆ , ⋆ Adarga, † University of Edinburgh, ‡ University of Exeter, Correspondence: lardet[dot]pierre[at]gmail.com
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-03276

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Topic Segmentation Using Generative Language Models Pierre Mackenzie ⋆,† , Maya Shah ⋆,‡ , Patrick Frenett ⋆ , ⋆ Adarga, † University of Edinburgh, ‡ University of Exeter, Correspondence: lardet[dot]pierre[at]gmail.com Abstract Topic segmentation using generative Large Language Models (LLMs) remains relatively unexplored. Previous methods use semantic similarity between sentences, but such models lack the long range dependencies and vast knowledge found in LLMs. In this work, we propose an overlapping and recursive prompting strategy using sentence enumeration. We also support the adoption of the boundary similarity evaluation metric. Results show that LLMs can be more effective segmenters than existing methods, but issues remain to be solved before they can be relied upon for topic segmentation. 1 Introduction Topic segmentation is the problem of dividing a string of text into constituent ‘segments’. Each segment should be semantically self-contained such that it is about one thing. For this work, segment boundaries always lie on sentence boundaries. We can then interpret segmentation as a binary classification task; given a list of input sentences, the model must decide whether there exists a boundary between each pair of adjacent sentences. Despite advances in LLMs, topic segmentation remains a relevant task. Information retrieval, long-document summarisation, classification and RAG 13 can all benefit from their inputs first being broken down by topic. For many open-source or resource-constrained models, context windows limit the size of input. Although newer models have very long context windows, 14 show that LLMs do not fully utilise long context. Segmentation can also be important for its own sake in dividing a document into constituent parts, to create a contents page, or summaries and titles for each section. Segmentation is a non-trivial task. The ambiguous definition of a segment leads to disagreements between humans annotators on where the ‘correct’ boundaries lie ( 10 ) . This is perhaps why there are few datasets in the field and none with human annotations of passages of text. Instead, annotations are normally derived from concatenations or metadata. For a machine learning model to attain performance comparable to humans is a daunting task that is hard to measure. Figure 1: The overlapping and recursive prompting strategy for segmentation. In (a), a long document is split into overlapping sections with sentence boundaries enumerated. In (b), each section is segmented by the LLM. In (c), the segments are joined. Finally, in (d), each segment is validated to ensure it is not too long or too short. 2 Related Work An influential framework introduced by ( 10 ) involves computing lexical similarity scores between adjacent sentences before boundaries are placed where similarity is lowest ( 8 ; 5 ) . Such a framework is still in use today but with semantic similarity calculated from embeddings. Different neural architectures have seen use such as RNNs ( 19 ; 12 ; 1 ) and attention-based models 16 ; 9 ; 15 . However, these models do not leverage the vast knowledge contained in the largest pre-trained language models. LLMs are the state of the art for a variety of NLP tasks. However, there has been little research into their use for topic segmentation. A loss-based approach was proposed by ( 6 ) in which boundaries are placed at peaks in the mean negative log likelihood of tokens in a sentence. This method relies on the dubious assumption that all information about segment boundary location can be expressed by the next token prediction loss. Due to resource constraints, we would have to use less powerful LLMs to test this method, and 20 finds that prompting LLMs outperforms this method. Therefore, we do not consider it here. Previous unpublished work has explored segmentation by prompting LLMs ( 20 ) . They find that prompting ChatGPT is the best dialogue segmentation model unless the input exceeds ChatGPT’s limit. Two prompting methods are proposed: one which asks the LLM to return the original text with characters delimiting boundaries and a second in which the LLM is asked to return semantic coherence scores for each pair of sentences in the range (0,1) . The first method does not satisfy a guarantee that the model will return the original document unedited and is massively wasteful of tokens. In the second method, the returned scores have no guarantee of directly corresponding to semantic coherence specifically for topic segmentation. Due to monetary constraints, we were unable to test more than one prompting method, and therefore we do not empirically compare against this work. We propose a new prompting method which ensures the output text is unedited, is vastly more token-efficient, and is not limited by the context window of the model. We show that prompting method outperforms existing non-prompting approaches by comparison with semantic similarity calculated on SentenceBERT ( 18 ) embeddings. boundary similarity ( n=2 ) boundary precision and recall Human Wiki Conc-Wiki Synthetic Human Wiki Conc-Wiki BP BR BP BR BP BR GPT3.5 0.38 0.25 0.29 0.35 0.51 0.60 0.36 0.55 0.42 0.63 FlanT5 0.25 0.24 0.41 0.33 0.38 0.46 0.43 0.37 0.65 0.63 BERTGraph 0.20 0.15 0.45 0.21 0.47 0.25 0.39 0.21 0.79 0.54 BERT 0.18 0.09 0.46 0.18 0.33 0.39 0.23 0.37 0.91 0.50 Split5 0.13 0.19 0.19 0.23 0.17 0.48 0.33 0.39 0.25 0.52 Table 1: Mean boundary similarity, precision (BP) and recall (BR) with n=2 for each model computed on a maximum of 150 documents per dataset. Note that Human has only 10 documents and Synthetic annotations were generated by ChatGPT. Best results for each metric/dataset are in bold. The LLMs perform better on all datasets except Conc-Wiki . They have better recall whereas precision varies per dataset. 3 Method 3.1 Datasets Human : We use a small dataset of 10 documents manually segmented by the authors. It is comprised of miscellaneous non-fiction articles. This was intended to provide high quality examples for qualitative analysis. Wiki : A wikipedia scrape 1 1 1 https://www.kaggle.com/datasets/ltcmdrdata/plain-text-wikipedia-202011/data was automatically segmented based on headings and filtered to remove articles with too few segments (<4), too short segments (<20 words) or too many artefacts (> 20% non-alphabetic characters), leaving ~1000 articles. We evaluate without headings present. Conc-Wiki : We randomly sampled segments from Wiki and concatenated them to form new incoherent articles, with segments drawn from completely different domains, leading to ~500 articles. Synthetic : Segmentations were generated by GPT-3.5 ( 4.1 ) on proprietary source data consisting of technical/news reports. This dataset was used for both fine-tuning FlanT5 ( 4.1 ) and evaluation. 3.2 Evaluation We follow the work of 7 who propose the ‘boundary similarity’ metric and associated precision/recall. The metric pairs segment boundaries between a hypothesised and references segmentation. Exact matches score 1 and no match scores 0, whilst matches within a distance n score linearly in the distance. Boundary similarity (B) is the mean score, while boundary precision/recall (BP/BR) are the mean score of matched hypothesis/reference boundaries, respectively. For further justification for the use of boundary similarity as opposed to more traditional metrics such as WD and Pk 17 , see the work of 7 , or our own investigations 2 2 2 redacted to retain anonymity . 3.3 LLM-Based Text Segmentation How can we get LLMs to output segment boundaries? We might consider prompting with

Read the complete text