Controlling Summarization Length Through EOS Token Weighting

Emmanouil Stergiadis, Zeno Belligoli, Eran Fainman, Ilya Gusev · 2025

Read Controlling Summarization Length Through EOS Token Weighting by Emmanouil Stergiadis, Zeno Belligoli, Eran Fainman, Ilya Gusev (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-05017.txt.

Introduction

Zeno Belligoli and co-authors offer a simple trick to control summary length in a 2025 paper. Many tasks need summaries of a set size, but past controls required surgery on the model, which breaks fit with ready-made models. A lighter method would travel better across systems.

Their method raises the weight of the end-of-sequence token during training, teaching the model to stop sooner or later on cue. Because it only changes the training loss, it works with any model shape and any decoding scheme. It also stacks with other control methods rather than replacing them. Tests check how closely output length follows the target. Tool builders get a cheap dial for length that they can add to existing summarizers.

Catalog record

CREATOREmmanouil Stergiadis, Zeno Belligoli, Eran Fainman, Ilya Gusev
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE23 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-05017

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Controlling Summarization Length Through EOS Token Weighting Emmanouil Stergiadis, Zeno Belligoli, Eran Fainman, Ilya Gusev Abstract Controlling the length of generated text can be crucial in various text-generation tasks, including summarization. Existing methods often require complex model alterations, limiting compatibility with pre-trained models. We address these limitations by developing a simple approach for controlling the length of automatic text summaries by increasing the importance of correctly predicting the E ​ O ​ S EOS token in the cross-entropy loss computation. The proposed methodology is agnostic to architecture and decoding algorithms and orthogonal to other inference-time techniques to control the generation length. We tested it with encoder-decoder and modern GPT-style LLMs, and show that this method can control generation length, often without affecting the quality of the summary. 1 Introduction Text summarization is the task of condensing essential information from a long text into a shorter one. Extractive text summarization methods create summaries by taking the most representative sentences from the original text, whereas abstractive text summarization focuses on generating completely new text [ 27 ] . This task finds applications in various domains, such as news [ 9 ] , scientific papers [ 17 ] , conversations [ 5 ] , and review [ 10 ] summarization. Summarization tasks tend to be accompanied by various constraints, often dictated by the requirements of an application or product. Examples of these constraints are capping the maximum length of the generated text, using specific keywords in the summary, following a specific format or style [ 4 ] . Furthermore, despite the rise of large language models like ChatGPT or GPT-4 [ 21 ] , we speculate (and confirm in Section 5 ) that simpler models can offer comparable summarization quality at a lower cost, making research in this field still relevant. In this work, we focus on controlling length in abstractive text summarization. This problem is motivated by the need to meet interface requirements, such as element sizes in mobile applications. In this context, summaries need to be of a desired character length to fit into the page to optimize user experience. To address this problem, we introduce a simple method of controlling summary length which involves weighting the end-of-sentence ( EOS ) token more than other tokens at training time. Intuitively, this allows the model to focus on correctly predicting when to stop the generation, thus inducing it to respect the summary length distribution in its training data. We conduct experiments on two model families and multiple decoding strategies to show the portability of the proposed approach across architectures and its complementarity to other inference-time length controlling techniques. 2 Previous work Methods for controlling the length of generated text can be categorized into two groups: learning-based and decoding-based approaches. Although learning-based methods involve alterations to the training architecture or loss function, decoding-based methods operate during the inference phase. Decoding-based techniques often involve preventing the model from producing the EOS token by assigning it a probability of negative infinity and truncating the text once the desired token count is achieved [ 23 ] , or by incorporating a length penalty into the beam-search decoding algorithm [ 20 ] . On the other hand, learning-based methods adapt the attention mechanism to be more sensitive to length [ 28 , 15 ] or train specialized embeddings that factor in the desired length of the generated text [ 12 , 3 , 14 , 25 ] . In addition, Makino et al. 2019 designed a modification of the objective function that increases the effectiveness of embedding-based methods, thus showing that the modifications of the training architecture and of the objective function are complementary to each other. Many of these techniques, however, entail intricate implementation steps and necessitate training new models from scratch, making them less feasible for integration with pretrained models. Notable exceptions to this constraint are the work of Miculicich et al. 2023 , who fine-tuned a pre-trained model with reversed positional encodings and showed competitive results both in terms of summary quality and length, and that of Chan et al. 2021 and Jie et al. 2023 who used a Markov decision process and reinforcement learning, respectively, to control the generation length. In line with this research trajectory, our method can be applied to train a new model from scratch as well as to fine-tune pretrained models. We refrain from altering the underlying architecture, and instead adopt a straightforward modification of the objective function which enhances our ability to govern generation length. 3 Methodology The intuition behind our method lies in the special importance of the EOS token during training. We note that the cross-entropy loss calculated on that particular token is the only loss component directly teaching the model to respect the summary length distribution in its training data 2 2 2 This is not the case when the summary consists of a single sentence terminated by a period. In that case, the period token acts as an extra, albeit somewhat weaker, EOS token. . During the computation of the loss, the signal from that particular token gets diluted by the averaging operation among all other generated tokens, which depending on the dataset can range in number from a few dozens to a few hundreds. We therefore hypothesize that simply boosting the weight of that loss component will help the model follow the training length distribution more closely, without significantly affecting overall performance. To be precise, our work aims at enforcing an upper bound for the generation length, which is why we are only interested in penalising false negatives when predicting EOS token (the loss component when the ground truth is EOS ). The exact weight to be applied is a hyper-parameter on which we run an ablation study. In formal terms, we start from the original form of the cross entropy loss calculated over the sequence: L_{1}=-\frac{1}{N}\sum_{n=1}^{N}\log\frac{e^{x_{n}^{y_{n}}}}{\sum_{v=1}^{|V|}e^{x_{n}^{v}}} (1) where V is the vocabulary, N is the sequence length, y_{n} the ground truth token at time-step n\in(1,N) and x_{n}^{v} is the logit for token v\in V at time-step n . We then add a weighting term to derive: L_{2}=-\frac{R}{N}\sum_{n=1}^{N}w_{y_{n}}\log\frac{e^{x_{n}^{y_{n}}}}{\sum_{v=1}^{|V|}e^{x_{n}^{v}}} (2) where w_{y_{n}}=\begin{cases}W,&\text{if }y_{n}=[EOS]\\ 1,&\text{otherwise}\end{cases} Because this weighting marginally impacts the norm of the loss and therefore its gradient, we apply a re-scaling factor R=\frac{N}{N+W-1} (3) to make sure the norm of weight updates are not impacted in expectation. This effectively changes the mean pooling of loss components 1/N in the original loss to 1/(N+W-1) which represents a loss computed over N+W-1 components: the N-1 non EOS ones and the EOS one counted W times. The weight of the EOS token W is a hyper-parameter that controls the balance between semantics and length: when W=1 , L_{2} goes back to treating EOS just as another token ( L_{1}=L_{2} ); as W\to\infty the loss assigns higher importance to not missing the EOS token, thus making its predicted sequences increasingly short (potentially at the expense of quality). 4 Experiments We devised two variants of the proposed methodology based on the availability of datasets with different characteristics. For this, we create subsets of CNN/Daily Mail Hermann et al. 2015a ; See et al. 2017 and XL-sum Hasan et al. 2021 with the desired characteristics. We consider only the

Read the complete text