Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models
Zihong Zhang, Liqi He, Zuchao Li, Lefei Zhang, Hai Zhao, Bo Du · 2025
Read Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models by Zihong Zhang, Liqi He, Zuchao Li, Lefei Zhang, Hai Zhao, Bo Du (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-19631.txt.
Introduction
Splitting text into words is a first step for language software, and for languages written without spaces it is a hard one. Zihong Zhang and five co-authors flip the usual order in a 2025 paper: instead of segmenting before understanding, they let large language models understand first and segment later. They test how well the models cut raw text in many languages using simple prompts.
Bigger models generally cut better. The team also builds a hybrid method pairing the models' insight with a classic fast pattern-matching machine, forming a word model that adapts to context. It beats traditional approaches, and the authors use the tests to probe what the big models truly grasp.
Catalog record
| CREATOR | Zihong Zhang, Liqi He, Zuchao Li, Lefei Zhang, Hai Zhao, Bo Du |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-19631
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Segment First or Comprehend First? Explore the Limit of Unsupervised Word Segmentation with Large Language Models Zihong Zhang, Liqi He, Zuchao Li, Lefei Zhang, Hai Zhao, Bo Du Abstract Word segmentation stands as a cornerstone of Natural Language Processing (NLP). Based on the concept of "comprehend first, segment later", we propose a new framework to explore the limit of unsupervised word segmentation with Large Language Models (LLMs) and evaluate the semantic understanding capabilities of LLMs based on word segmentation. We employ current mainstream LLMs to perform word segmentation across multiple languages to assess LLMs’ "comprehension". Our findings reveal that LLMs are capable of following simple prompts to segment raw text into words. There is a trend suggesting that models with more parameters tend to perform better on multiple languages. Additionally, we introduce a novel unsupervised method, termed LLACA ( L arge L anguage Model-Inspired A ho- C orasick A utomaton). Leveraging the advanced pattern recognition capabilities of Aho-Corasick automata, LLACA innovatively combines these with the deep insights of well-pretrained LLMs. This approach not only enables the construction of a dynamic n n -gram model that adjusts based on contextual information but also integrates the nuanced understanding of LLMs, offering significant improvements over traditional methods. Our source code is available at https://github.com/hkr04/LLACA 1 Introduction Figure 1: Examples of LLM-WS Understanding language is the core task of NLP Zhang et al. (2019) . To measure the understanding capabilities of language models, various Natural Language Understanding (NLU) tasks have been proposed, such as Question Answering Namazifar et al. (2021) ; Yao et al. (2023) ; He et al. (2024) and Sentiment Classification Arevalillo-Herráez et al. (2022) ; Jing et al. (2021) . However, most of these tasks only assess the language model’s understanding of the overall meaning of sentences, lacking an evaluation of the language model’s understanding capabilities at a fine-grained level. In many languages, such as Chinese and Japanese, there are no explicit word boundaries. Therefore, word segmentation is a crucial foundational step in NLP tasks like syntactic analysis Cereda et al. (2018) ; Li et al. (2018) , information retrieval Ponte and Croft (2017) , and machine translation Moslem (2024) ; Li et al. (2020) for these languages. Most previous research on word segment has adhered to the principle of "segment first, comprehend later", because word segment has long been regarded as the first step in NLP Zhao et al. (2006) . However, the human brain’s process of analyzing sentences typically involves an interactive process of segmentation and comprehension, where segmentation depends on comprehension and vice versa, especially in sentences with ambiguity. Therefore, word segmentation can also be the last step in NLP, that is, to test the understanding capabilities of a language model. Large Language Models (LLMs) are the best linguists. The emergence of LLMs has marked significant advancements in NLU. And the capabilities of LLMs are no longer based on word segmentation. However, word segmentation can serve as an indicator to effectively evaluate the semantic understanding capabilities of LLMs. Additionally, we can utilize LLMs for word segmentation, and we propose a word segment framework named LLM-Word Segmentation (LLM-WS). We employ the framework to explore the limit of unsupervised word segmentation with LLMs and evaluate the semantic understanding capabilities of LLMs based on word segmentation. Our research demonstrates that LLMs can perform word segmentation on raw text based on simple prompts, as shown in Figure 1 . In the upper section of Figure 1 , we demonstrate how LLMs can accurately segment typical Chinese sentences with ambiguity by taking context into account. Based on the different scenarios of "charity gala" and "store", the two identical Chinese word sequences "ping pong balls are auctioned" and "ping pong racket is sold" were correctly segmented by LLMs. The example in the bottom left corner of Figure 1 illustrates that LLMs can correctly segment consecutive identical characters within a sentence. The example in the bottom right corner of Figure 1 illustrates that LLMs can segment ambiguous sentences by considering the likelihood of different interpretations within the actual context. Undoubtedly, "father enters the room" is more common than "father enters the bag". The aforementioned examples also demonstrate that LLMs possess capabilities of word segmentation across multiple languages, including Chinese, Japanese, and Korean. Figure 2: Evaluations on datasets MSR (Simplified Chinese), KWDLC (Japanese) and BEST (Thai). In previous word segmentation research, supervised and unsupervised methods are two main learning paradigms for word segmentation. While supervised methods have indeed demonstrated remarkable efficacy Zhao et al. (2019) , they encounter hurdles such as a heavy reliance on a great deal of manually labeled corpora and poor domain adaptability. Unsupervised word segmentation approaches can avoid the need for a large amount of human labor required for labeled datasets. Additionally, previous research has demonstrated that unsupervised word segmentation methods have better stability for unseen words and adaptability to new domains. Previous unsupervised methods can be broadly classified into two types: discriminative models and generative models. The former evaluates the quality of word candidates using carefully designed goodness measures. However, this method lacks the capability to handle ambiguous strings, which is a major source of segmentation errors. The latter designs models to find the optimal segmentation with the highest generative probability. These methods have better stability in segmenting ambiguous sentences. With the rapid development of neural networks in recent years, research has been conducted to perform unsupervised word segmentation using neural generative models Wang et al. (2017) ; Sun and Deng (2018) , achieving competitive performance to the state-of-the-art statistical models. The most significant difference between our approach and previous research is the implementation of comprehension-based word segmentation. This represents a new era in the development of word segmentation methods. Based on LLMs trained on massive corpora, our framework named LLM-WS can explore the limit of unsupervised word segmentation. Figure 3: The illustration of an Aho-Corasick automaton with patterns "she", "he", and "her", with final states in double circles and failure links in red. Specifically, in LLM-WS, we propose an unsupervised word segmentation method named LLACA ( L arge L anguage Model-Inspired A ho- C orasick A utomaton). Leveraging the profound insights into language provided by well-pretrained LLMs, we utilize the advanced pattern recognition capabilities of Aho-Corasick automata Aho and Corasick (1975) to achieve unsupervised word segmentation. We utilize LLMs to conduct word segmentation within the corpus and compute word frequencies. Subsequently, we integrate the computational outcomes derived from LLMs with the AC automaton to facilitate efficient unsupervised word segmentation. Our findings reveal that larger language models often exhibit stronger segmentation capabilities. Conversely, models with fewer parameters frequently struggle to adhere to segmentation prompts, leading to significant hallucinations and deviations from human-annotated gold standards. Notably, the Chinese LLM Qwen1.5-7B-Chat has already surpassed previous state-of-the-art results on the Chinese word segmentation tasks for the MSR and PKU datasets. 2 Related Work For word segmentation, the simplest but effective method is the maximum matching model Jurafsky and Martin (2014) . Beginning at the start of a string, the maximum matching model selects the longest dictionary word that corresponds to the current position and then moves forward to the end of that matched word within the string. However, it is clear that this method