LLM-based Affective Text Generation Quality Based on Different Quantization Values
Yarik Menchaca Resendiz, Roman Klinger · 2025
Read LLM-based Affective Text Generation Quality Based on Different Quantization Values by Yarik Menchaca Resendiz, Roman Klinger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-19317.txt.
Introduction
Yarik Menchaca Resendiz and Roman Klinger test, in a 2025 paper, what shrinking a language model does to the feeling in the text it writes. Models can now write warm, human-like prose, but they are huge and costly to run. Cutting the number of bits used to store their weights saves memory, at some cost to quality, and the pair measure that trade for emotional text.
They ran five open models at eight, sixteen, and thirty-two bits of precision, using an emotion classifier and ten seed prompts to grade the output. Bit cuts saved up to seventy-six percent of memory. Scores fell by up to ten points for larger models but rose for smaller ones, while run time roughly doubled. Bigger models at low precision often beat smaller ones at high precision for the same memory.
Catalog record
| CREATOR | Yarik Menchaca Resendiz, Roman Klinger |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-19317
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
LLM-based Affective Text Generation Quality Based on Different Quantization Values Yarik Menchaca Resendiz, Roman Klinger Abstract Large language models exhibit a remarkable capacity in language generation and comprehension. These advances enable AI systems to produce more human-like and emotionally engaging text. However, these models rely on a large number of parameters, requiring significant computational resources for training and inference. In some scenarios, accessing these resources can be challenging (e.g., budget or hardware limitations). Techniques like reducing precision bits can make models more memory-efficient, reducing the computational resources needed, at the cost of reduced accuracy. This paper addresses the trade-off between different quantization values, GPU RAM utilization, and text quality in affective text generation (e.g., “I really enjoy running in the snow-covered forest”). To evaluate, we use an emotion classifier and ten seed prompts to generate affective text. We test three setups of precision bits (8, 16, and 32) across five open-weight language models from two different families. Our findings demonstrate that bit reductions lead to memory savings, achieving a reduction of 76 %. However, this optimization comes with a trade-off, leading to a decrease of up to 10 pp in F 1 score for larger models and an increase of 10 pp for smaller models, along with roughly double the inference time. In terms of text quality, larger models at lower quantization levels generally outperform smaller, higher-precision models – while requiring similar memory. 1 Introduction Large Language Models ( Touvron et al., 2023 ; Radford et al., 2018 ; Brown et al., 2020 ; Radford et al., 2019 , LLMs,) have demonstrated unprecedented capabilities in language generation and contextual comprehension. These advancements have improved the ability of AI systems (e.g., virtual assistants, customer service bots), to generate human-like and emotionally engaging text. This ability to produce affective text is crucial for making AI systems capable of building user trust and satisfaction. However, the substantial computational resources required by these models often make them inaccessible to individuals or organizations with limited resources. A democratizing solution is quantization, which involves converting high-precision values to lower-precision ones (e.g., from 32 to 8 bits) to reduce overall model size ( Hu et al., 2022 ; Dettmers et al., 2023 ) . This approach makes LLMs more memory-efficient, requiring less GPU memory, storage, and energy. However, the primary drawback of quantization is the potential loss of accuracy, with higher bit reductions posing a greater risk. This reduction in precision sets the stage for exploring the trade-off between computational resources, such as GPU RAM, inference time, and text quality. Our research aims to answer: What is the trade-off between precision in bits on LLMs in affective text generation? Post-training quantization is popular due to its simplicity and effectiveness. Jacob et al. (2018) demonstrated that it can reduce full-precision models to 8-bit integers with minimal accuracy loss, making it suitable for use on edge devices (e.g., smartphones, tablets). Quantization-aware training incorporates quantization during training, producing models more robust to precision reduction. Courbariaux et al. (2015) introduced BinaryConnect, a quantization-aware method that trains neural networks with binary weights, significantly reducing memory usage and computational complexity. Quantization methods are essential for complex tasks such as emotion-conditioned text generation, which enhances human-computer interaction but demands significant computational resources. Most research focuses on sentiment polarity ( Zhang et al., 2019 ; Maqsud, 2015 ; Niu and Bansal, 2018 ) , with fewer studies on emotion conditioning Ghosh et al. (2017) ; Song et al. (2019) ; Zhou et al. (2018) ; Cui et al. (2022) ; Colombo et al. (2019) ; Menchaca Resendiz and Klinger (2023) . None of the previous work has focused on the trade-off between affective text generation and quantization, which is crucial for enabling AI assistants to operate on devices with limited resources (e.g., smartphones, and laptops). Our research addresses this gap by investigating the trade-offs between GPU RAM, inference time, and text quality when using different precision bits in affective text generation. 2 Methodology The presented work aims to provide an understanding of affective text generation using multiple precision bits 1 1 1 The code can be found at Blinded for review. . Section 2.1 introduces the seed prompts used as affective generation prompts for the LLMs. Section 2.2 describes the LLMs and the quantization configurations. Section 2.4 explains the automatic evaluation setup. Finally, Section 2.5 presents the text quality evaluation. 2.1 Seed Prompts We select a set of 10 seed prompts strategically designed to guide the language models for effective affective text generation. This collection comprises a mix of straightforward prompts (e.g., Generate a text that expresses Emotion , Phrases that express Emotion ) that users can easily conceive, alongside prompts drawn from psychological studies, such as the ISEAR survey (e.g., "Please describe a situation or event—in as much detail as possible—in which a reader felt Emotion ?"). The following diverse combination ensures a comprehensive exploration of emotional expressions, encompassing both intuitive and scientifically validated inputs. 1. Describe a situation where a person felt Emotion . 2. Write a text that expresses Emotion . 3. Phrases that express Emotion . 4. What is a sentence example for Emotion ? 5. Can you provide an example of a situation where someone experienced Emotion ? 6. What is an example of a Emotion sentence? 7. Emotion sentence. 8. Experience for Emotion ? 9. Please describe a situation or event—in as much detail as possible—in which a reader felt Emotion . 10. Please complete the sentence: I felt Emotion when/because. 2.2 Precision Bits To address the research question, What is the trade-off between precision in bits on LLMs in affective text generation? , we utilize Llama 2 Chat ( Touvron et al., 2023 ) with three parameter sizes (7B, 13B, and 70B) and Mistral 7B Jiang et al. (2023) and Mixtral-8x7B ( Jiang et al., 2024 ) . Each model is loaded into memory using three different precisions: 8, 16, and 32 bits – the default setting. We employ a total of 15 configurations to generate affective text from the seed prompts. 2.3 Text Generation The text is generated using the LLMs and the 10 seed prompts. We design an Emotion-Prompt that integrates the seed prompt (for instance, “Write a text that expresses Emotion .”) with one of seven specified emotions: Joy, Fear, Anger, Sadness, Disgust, Shame, and Guilt. In this setup, Emotion is replaced by each emotion category (see Table 5 in appendix B , for examples). For every Emotion-Prompt , we employ a beam search strategy to generate the three most likely sentences – 210 sentences in total. This process uses a beam size of 30, sets the next-token temperature to 0.7, and applies a top-p (nucleus) sampling strategy at 0.7. Additionally, we take measures to ensure the generated output is free from sentences that contain duplicate instances of any bigram. The inference tasks for the LLMs models are performed and evenly distributed across a GPU cluster equipped with 8 NVIDIA RTX A6000 GPUs, for 10 times per model configuration. 2.4 Automatic Classification To evaluate the performance of the LLMs configuration automatically, we use an emotion classifier fine-tuned on the ISEAR dataset Scherer and Wallbott (1994) . The emotion classifier is built on top of RoBERTa