Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback
Tom S. Juzek, Zina B. Ward · 2025
Read Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback by Tom S. Juzek, Zina B. Ward (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-01930.txt.
Introduction
Tom Juzek and Zina Ward investigate why large language models overuse certain words, and whether training from human feedback is the cause. Their paper came out in 2025. Such feedback trains a model on which of two answers people prefer.
They first give a simple procedure for detecting word preferences that this training may have induced. They then reran a small version of the feedback process with human participants. Participants systematically preferred text variants containing certain words, which reproduces the overuse effect. The authors frame this as a mild misalignment, a gap between what trainers reward and what users expect. They argue for more transparency about both training data and procedures.
Catalog record
| CREATOR | Tom S. Juzek, Zina B. Ward |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 34 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-01930
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback Tom S. Juzek, Zina B. Ward Abstract Large Language Models (LLMs) are known to overuse certain terms like “delve” and “intricate.” The exact reasons for these lexical choices, however, have been unclear. Using Meta’s Llama model, this study investigates the contribution of Learning from Human Feedback (LHF), under which we subsume Reinforcement Learning from Human Feedback and Direct Preference Optimization. We present a straightforward procedure for detecting the lexical preferences of LLMs that are potentially LHF-induced. Next, we more conclusively link LHF to lexical overuse by experimentally emulating the LHF procedure and demonstrating that participants systematically prefer text variants that include certain words. This lexical overuse can be seen as a sort of misalignment, though our study highlights the potential divergence between the lexical expectations of different populations – namely LHF workers versus LLM users. Our work contributes to the growing body of research on explainable artificial intelligence and emphasizes the importance of both data and procedural transparency in alignment research. 1 Introduction Following the arrival of Large Language Models (LLMs), observers were quick to note their tendency to overproduce certain lexical entries [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 ] . Much of the discourse centered on Scientific and academic English, focusing on words such as “delve”, “intricate”, and “realm.” For this reason, we also concentrate on Scientific English here. While changes in Scientific English over decades and centuries are well-documented [ 10 , 11 , 12 , 13 ] , the language shifts following the introduction of LLMs have been unprecedented, with certain words (like “delve”) seeing a sudden and dramatic increase in usage. Figure 1: An illustration of the procedure used to identify lexical preferences that are potentially induced by Learning from Human Feedback (LHF); created with Canva. Thus, it has been established that certain lexical biases exist in LLMs, with evidence demonstrating their influence on written language. However, the question of why this lexical overrepresentation arises remains open. While some have pointed to Learning from Human Feedback (LHF) as a significant contributor to these lexical choices [ 14 , 15 ] , conclusive evidence is still missing. Learning from Human Feedback is a procedure applied after initial model training during which human evaluators indicate preferences through A/B testing or ranking. It was first introduced in the form of Reinforcement Learning from Human Feedback (RLHF; [ 16 , 17 ] ), though a more recent and increasingly popular form of LHF is Direct Preference Optimization (DPO), which aligns models by directly optimizing for human preferences without relying on reinforcement learning [ 18 ] . LHF was introduced to align models more closely with human preferences. Alignment, which reflects “how closely the model’s opinions or stances mirror those of different social groups” [ 19 ] , is a major challenge in AI [ 20 , 21 , 22 ] . A model is misaligned for a target group when its output does not align with the group’s opinions, values, and/or expectations. LHF is recognized as a key factor contributing to the success of models like ChatGPT [ 23 ] . However, researching the effects of LHF is difficult due to lack of transparency surrounding the procedures and datasets used in model development [ 24 ] . The present study addresses the potential link between LHF and the lexical choices of LLMs through a two-step process. First, we introduce a method for identifying lexical preferences in LLMs that are potentially induced by LHF. This procedure can aid efforts to mitigate the most extreme cases of lexical overrepresentation (Section 2 ). Second, we conduct an experiment that emulates the LHF procedure in order to test whether humans indeed prefer texts containing the words identified by our initial procedure. This represents an empirical test of the hypothesis that LHF plays a role in shaping LLMs’ lexical choices (Section 3 ), based on one of Meta’s popular Llama models. While our findings provide evidence for an LHF effect, other contributing factors remain to be systematically investigated. Finally, we discuss the implications of our study (Section 4 ) and its limitations (Section 5 ). Related Work Many studies explore the linguistic behavior of LLMs and their effects on (written) human language [ 1 , 2 , 3 , 4 , 5 , 6 , 7 , 8 ] , with a few investigating spoken language [ 25 , 26 ] . Most of this work is situated at the word level, though there is also research on syntactic behavior [ 27 ] . Procedures for identifying LHF-induced overuse have been proposed [ 5 , 9 ] , but these involve a manual component [ 5 , 9 ] and/or have a different focus [ 5 ] . Similar concerns have been raised for other behaviors exhibited by LLMs (see the xAI literature; [ 28 , 29 , 30 ] ). Overlap between human linguistic preferences and model behavior has been shown, though with a small sample [ 9 ] . For non-lexical form (such as boldface or emoji use), it has been found that even subtle differences in preferences during human preference training can result in substantial differences in model behavior [ 31 ] . 2 Procedure to Identify Potentially LHF-Induced Lexical Preferences As a first step, we develop a low-cost procedure to identify lexical preferences in LLMs that may originate from LHF training. Our approach involves generating language outputs from both a pre-LHF model and a post-LHF model and then comparing word usage in the outputs. Here, we use Llama 3.2-3B Base and Llama 3.2-3B Instruct [ 33 ] (via the Hugging Face Transformers library [ 34 ] ). The Llama family is, to our knowledge, the closest available approximation between models trained with and without LHF, which for Llama 3 involves Direct Preference Optimization. At the time of our research, Llama 3.2 was the most recent version of the Llama model family. While larger variants (11B and 90B) were available, they primarily added multimodal capabilities; improvements in textual reasoning abilities were minor [ 35 ] . At the time of research, a broader model comparison would have been difficult: of all the major LLM developers, only Meta had released both base and instruction-tuned models. Since then, models like OLMo [ 36 ] and Falcon [ 37 ] have gained popularity and would now be strong options. There are other differences between Llama Base and Llama Instruct [ 33 ] , most notably instruction tuning, optimization for tooling, and safety mitigation. However, none of these, including instruction tuning [ 38 ] , are known to contribute to lexical overrepresentation. LHF remains the most plausible contributor to shifts in language output. This makes the Llama models well-suited for our purposes. All technical implementations described in this paper were carried out in Python 3 ( [ 39 ] ; v3.12.3). Although our study focuses on Scientific English, the procedure we present is transferable to other domains. Here the procedure is applied to abstracts from PubMed from 2020 [ 40 ] , as this predates the mainstream availability of