Advancing Sequential Numerical Prediction in Autoregressive Models
Xiang Fei Jinghui Lu Qi Sun Hao Feng Yanjie Wang, Wei Shi An-lan Wang Jingqun Tang Can Huang · 2025
Read Advancing Sequential Numerical Prediction in Autoregressive Models by Xiang Fei Jinghui Lu Qi Sun Hao Feng Yanjie Wang, Wei Shi An-lan Wang Jingqun Tang Can Huang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13077.txt.
Introduction
Xiang Fei and co-authors fix a blind spot in how language models handle numbers in a 2025 paper. Models read text token by token, and standard training treats each digit as an isolated symbol. That ignores the plain fact that nine sits next to ten, so number sequences come out shaky.
Their loss function works at two levels. At the digit level it charges more for guesses that land far from the true value, using a distance borrowed from earth-moving math. At the sequence level it penalizes the whole predicted number's gap from the target. Tests across tasks showed clear gains, and the loss plugs into existing large models without redesign.
Catalog record
| CREATOR | Xiang Fei Jinghui Lu Qi Sun Hao Feng Yanjie Wang, Wei Shi An-lan Wang Jingqun Tang Can Huang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13077
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Advancing Sequential Numerical Prediction in Autoregressive Models Xiang Fei Jinghui Lu Qi Sun Hao Feng Yanjie Wang, Wei Shi An-lan Wang Jingqun Tang Can Huang Abstract Autoregressive models have become the de facto choice for sequence generation tasks, but standard approaches treat digits as independent tokens and apply cross-entropy loss, overlooking the coherent structure of numerical sequences. This paper introduces N umerical T oken I ntegrity Loss (NTIL) to address this gap. NTIL operates at two levels: (1) token-level, where it extends the Earth Mover’s Distance (EMD) to preserve ordinal relationships between numerical values, and (2) sequence-level, where it penalizes the overall discrepancy between the predicted and actual sequences. This dual approach improves numerical prediction and integrates effectively with LLMs/MLLMs. Extensive experiments show significant performance improvements with NTIL. All resources are available at https://github.com/xfey/NTIL . 1 Introduction In recent years, sequence generation has become a crucial approach for implementing a broad range of AI applications, including visual question answering ( Wang et al., 2024d ; Reich and Schultz, 2024 ; Fan et al., 2024 ; Liu et al., 2024b ; He et al., 2025 ; Wang et al., 2025 ; Lu et al., 2024b ) , key information extraction ( Kim et al., 2024 ; Yu et al., 2024 ; Kang et al., 2024 ; Wang et al., 2024a ; Yi et al., 2025 ; Lu et al., 2023a ; Lu et al., 2024a ) , object detection ( Wen et al., 2024 ) , math reasoning Zhao et al. (2024) , text spotting ( Li et al., 2024 ; Yu et al., 2025 ) , and automatic audio recognition ( Zhou et al., 2024 ) . Figure 1: Sequence-level digit token loss illustration. Autoregressive models, especially large language models (LLMs) such as GPT Achiam et al. (2023) , LLaMA Touvron et al. (2023) ; Dubey et al. (2024) , Qwen Yang et al. (2024) ; Wang et al. (2024c) series, with multi-modal large language models (MLLMs) based on them, now dominate the sequence generation tasks. During training, these models generate sequences token-by-token, typically using cross-entropy (CE) loss, to minimize the negative log-likelihood of the ground truth token at each time step. However, CE loss has several inherent limitations when predicting numerical values. Specifically, CE suffers from Limitation 1 that it ignores the inherent closeness between numerical tokens, where each digit in a numerical prediction is not independent but related to its neighboring digits. For example, in Figure 2 (a) and 2 (b), for the ground truth token “3”, the CE loss yields same values of -log(0.5) for different prediction distributions. However, the distribution in Figure 2 (b) is more accurate, as it assigns higher probability to the neighboring token “2”. CE also suffers from Limitation 2 that it fails to capture the holistic numerical error when sequential tokens are involved, as it focuses on the precision of each token rather than the overall value . In an autoregressive generation manner, producing a numerical value typically requires consecutive time steps. For example, the target value “0.98” requires the prediction of four sequential tokens — “0” , “.” , “9” , “8” . Thus, a prediction such as 1.01 ( “1” , “.” , “0” , “1” ) incurs a high CE loss as the first, third and fourth tokens are significantly different from the target tokens. Conversely, a prediction like 1.98 ( “1” , “.” , “9” , “8” ) could yield a lower CE loss due to a closer match at the token level, despite the overall numerical difference being larger (1.00 vs. 0.03). This discrepancy shows the limitation of CE in evaluating predictions holistically. Figure 2: Cross-entropy fails to distinguish predictions, whereas EMD correlates smaller loss for better predicted distributions. To overcome the above issues, we introduce a novel sequence-level numerical prediction loss: N umerical T oken I ntegrity L oss (NTIL) . At the token-level, NTIL replaces the traditional CE loss with Earth Mover’s Distance (EMD) ( Rubner et al., 1998 ) . Additionally, we enhance the EMD with an Exponential Position-based Weighting scheme (Section 3.1 ), which leverages place-value number systems to better capture the nuanced differences between numerical distributions at each time step. At the sequence-level, NTIL evaluates the overall numerical difference between predicted and actual sequences through Multi-token Numerical Optimization (Section 3.2 ), considering all time steps holistically, as illustrated in Figure 1 . It enables NTIL to effectively model the actual value of digit sequences, and capture discrepancies across the consecutive numerical range, moving beyond simple token-by-token comparison. To the best of the authors’ knowledge, it is the first time that EMD is used as an optimization method for autoregressive models. Moreover, our holistic approach is the first of its kind to improve sequential numerical prediction by considering numerical tokens across multiple time steps. Our method can be seamlessly integrated into both LLMs and MLLMs. Experimental results show that NTIL boosts performance in tasks requiring precise numerical outputs, such as object detection, text spotting, and math reasoning (Section 4 ). 2 Related Work The Earth Mover’s Distance (EMD) measures the minimal cost of transforming one distribution into another, and has become a valuable metric in deep learning applications. Notably, Wasserstein GAN Arjovsky et al. (2017) uses EMD as its loss function to stabilize training in GANs. Cuturi (2013) and Courty et al. (2016) also adopted EMD for smoothing the training procedure. Despite the success of EMD, it has not been applied to autoregressive models. Most recently, autoregressive models, especially LLMs, have advanced NLP ( Radford et al., 2019 ; Touvron et al., 2023 ; Lu et al., 2023b ; Cui et al., 2025 ) , and multi-modal tasks ( Alayrac et al., 2022 ; Wang et al., 2024c ; Feng et al., 2025 ; Lu et al., 2025 ) . While the tasks mentioned above require high precision in numerical value prediction, none of the previous works have specifically optimized for this criterion. Our work addresses this gap by focusing on advancing the sequential numerical prediction for autoregressive models. 3 Method This section details the components of the proposed method. Section 3.1 proposes exponential weighted EMD to single digits; Section 3.2 describes how we go through multiple digital tokens to derive a simple yet effective numerical measure. 3.1 Exponential Position-Based Weighting For token-level prediction, to address Limitation 1 in Section 1 , we replace the conventional CE loss with EMD to account for the ordinal relationship during optimization. The preliminaries for both CE and EMD objectives, and the simplification via numerical prediction, are outlined in Appendix D . Furthermore, we extend EMD to account for the place-value number systems, where leading digits have greater numerical significance. We implement an exponential weighting scheme to progressively assign weights based on digit positions, to scale their contributions to the loss accordingly: \mathbf{W_{exp}}=\left[(1+\sigma)^{n-i-1}\right]_{i=0}^{n-1}, (1) where \sigma is the exponential increment rate, and n is the length of consecutive digits. This implementation helps the model understand the order relationship between consecutive numbers. 3.2 Multi-Token Numerical Optimization To overcome Limitation 2 outlined in Section 1 , we propose the following procedure and losses. Differentiable Numerical Value Construction. In this step, we construct the complete numerical value from consecutive discrete