Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models
Yu Wang ⋄ \diamond Leyi Lao ⋄ \diamond Langchu Huang, Gabriel Skantze Yang Xu ∗ * Hendrik Buschmeier ∗ * · 2025
Read Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models by Yu Wang ⋄ \diamond Leyi Lao ⋄ \diamond Langchu Huang, Gabriel Skantze Yang Xu ∗ * Hendrik Buschmeier ∗ * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-20237.txt.
Introduction
Small words like uh-huh and um steer everyday conversation, yet language models often treat them as noise. Yu Wang and five co-authors examine how fine-tuned models represent these backchannels and fillers in a 2025 arXiv study. They fine-tune models three different ways on dialogue corpora in English and Japanese. The corpora keep the small expressions and label them, which makes the study possible.
Clustering analysis showed that fine-tuned models separate the subtle meanings of different backchannels and fillers more clearly. Standard generation scores also rose after fine-tuning. Closer reading confirmed that model utterances sounded more human in their use of these expressions. The findings suggest a route from general language models toward conversational ones. Such models would handle the small social signals of dialogue with more care.
Catalog record
| CREATOR | Yu Wang ⋄ \diamond Leyi Lao ⋄ \diamond Langchu Huang, Gabriel Skantze Yang Xu ∗ * Hendrik Buschmeier ∗ * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 83 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-20237
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Investigating the Representation of Backchannels and Fillers in Fine-tuned Language Models Yu Wang ⋄ \diamond Leyi Lao ⋄ \diamond Langchu Huang, Gabriel Skantze Yang Xu ∗ * Hendrik Buschmeier ∗ * Abstract Backchannels and fillers are important linguistic expressions in dialogue, but are under-represented in modern transformer-based language models (LMs). Our work studies the representation of them in language models using three fine-tuning strategies. The models are trained on three dialogue corpora in English and Japanese, where backchannels and fillers are preserved and annotated, to investigate how fine-tuning can help LMs learn their representations. We first apply clustering analysis to the learnt representation of backchannels and fillers, and have found increased silhouette scores in representations from fine-tuned models, which suggests that fine-tuning enables LMs to distinguish the nuanced semantic variation in different backchannel and filler use. We also use natural language generation (NLG) metrics to confirm that the utterances generated by fine-tuned language models resemble human-produced utterances more closely. Our findings suggest the potentials of transforming general LMs into conversational LMs that are more capable of producing human-like languages adequately. 1 Introduction In our daily life, it is quite ubiquitous to encounter conversation such as below: Alice: Did you check your inbox? Bob: uh-huh . Alice: Good. Bob: uh , but why? The bold words in this dialogue are examples for backchannels ( ‘uh-huh’ ) and fillers ( ‘uh’ ), which play an important role in managing the flow of conversation. The use of backchannels/fillers is seen as an important way for interlocutors to negotiate the common ground ( Clark 1996 ) . Since backchannels/fillers usually do not convey information directly and have only pragmatic functions Jucker and Smith 1998 (e.g., expressing affirmation or disagreement to the previous utterance), they are semantically ‘bleached’ elements ( Fuller 2003 ) and deemed optional as they lack concrete meanings ( Schourup 1999 ; Fuller 2003 ) . As a result, the NLP community often treats them similarly to ‘stop words’, excluding them during pre-processing as a method to clean the data and improve accuracy ( Sarica and Luo 2021 , e.g., see) . For instance, in dependency parsing for spoken dialogue data, studies such as Charniak and Johnson 2001 , Jørgensen 2007 and Dobrovoljc and Martinc 2018 report that excluding backchannels/fillers such as ‘uh’ and ‘er’ in the Switchboard dataset ( Holliman et al. 1992 ) can significantly improve parsing accuracy. Although this has been a common practice, previous studies have highlighted that backchannels and fillers can contain rich contextual meaning in dialogue: backchannels such as ‘yeah’ , ‘okay’ , for example, can serve as feedback towards previous speech ( Clark 1996 , pp. 32) . Fillers, such as ‘uh’, ‘um’ , are used as signals of disfluency ( Rose 2015 ) and reflect the speaker’s cognitive processing when searching for the next word ( Clark and Fox Tree 2002 ) , thus indicating the speaker’s cognitive load (e.g., see Table 1 of Berthold and Jameson 1999 and Rose 2015 ). Thus, fillers can be regarded as an important signal to the listener that the speaker needs some time to complete the utterance (and wants to hold the turn; Ball 1975 ). Further studies using qualitative approaches show that backchannels/fillers, as a source of feedback, play an important role in the incremental updating of dialogue from a semantic perspective, controlling the flow of information during conversation ( Bergey and DeDeo 2024 ) and improving the mutual understanding between the interlocutors in order to reach a joint goal ( Hough et al. 2015 ; Howes and Eshghi 2021 ) . In order to figure out the potential of transforming language models into conversational language models, which can utilise backchannels/fillers, it is essential to establish how effectively they can be learnt and represented in language models (LMs). The obstacle for answering the question is that, as emphasized in Liesenfeld and Dingemanse 2022 and Dingemanse and Liesenfeld 2022 , most language data in NLP is text-based. Well-formatted and well-annotated interaction data such as conversation data, is often insufficient in size and quality. Consequently, there is limited contextual information on backchannels/fillers for LMs to learn from, and thus LM-based automatic speech recognition (ASR) systems will perform poorly in recognise turn taking/holding, which is largely moderated by backchannels and fillers. Moreover, due to the absence of backchannels and fillers during the pre-training phase, most pre-trained LMs have limited knowledge of these elements. Consequently, the content generated by these LMs tends to be text-like and distant from real dialogue in its form 1 1 1 In our opinion, this case is also applicable to LLM such as LLaMA-3-8B, as evidenced in our observation shown in the examples in Figure 6(a) to 6(d) in the Appendix, during the NLG task, the pre-trained LLM barely generates any backchannels and fillers. , which, as result, fails to ‘[put] natural in natural language processing’ ( Chrupała 2023 ) and lack the ability to act as competent dialogue agents that can, for example, produce (i) appropriate backchannels as feedback to user utterances, and are therefore not considered attentive enough to meet human user expectations ( Buschmeier and Kopp 2018 ) , and (ii) natural fillers in their utterances, which are vital for their role in organising speech and their communicative functions in spoken language understanding ( Dinkar et al. 2022 ) . As a result, an LM which cannot use backchannels/fillers properly cannot be used for building a competent conversational agent. Our work presented here therefore addresses backchannels/fillers as important linguistic phenomena that are not well reflected in language models and specifically investigates the issue of learning the representations of backchannels/fillers in language models. A solution that proved effective in tackling dialogue phenomena issue, is fine-tuning ( Noble and Maraev 2021 , see, e.g.,) . Although there has been a great deal of work attempting to answer the question of how the representation of linguistic knowledge by a language model, such as BERT ( Devlin et al. 2019 ) and GPT2 Radford et al. 2019 , is altered after fine-tuning, we believe our study makes a meaningful contribution, as an initial study trying to answer how and how well language models can represent backchannels/fillers after fine-tuning. Our general research question ( RQ1 ) and three more specific research questions are: RQ1 : Can modern transformer-based language models, such as BERT, GPT-2 and a larger language model such as LLaMA-3 8B, Qwen 3 8B learn representations for backchannels/fillers through fine-tuning? RQ2 : What role does contextual information play when we try to obtain the representations of backchannels/fillers from LMs? RQ3 : Which of the studied LMs can benefit more from fine-tuning? RQ4 : Do different fine-tuning strategies make a difference in learning the representation of backchannels/fillers? 2 Background 2.1 Backchannels/Fillers as Discourse Markers Backchannels/fillers are considered to be discourse marker Jucker and Smith 1998 ; Jucker 1998 . They are semantically ‘bleached’ elements in conversation, such as oh, yeah, uh, uhm which can be either fillers or backchannels – depending on whether they are within the dialogue as a sign of disfluency, or stand alone or at the beginning of an utterance, which are then usually taken as feedback to the previous utterance. As indicated