Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models
Hao Zhou, Guergana Savova, Lijing Wang · 2025
Read Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models by Hao Zhou, Guergana Savova, Lijing Wang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07329.txt.
Introduction
Nghia Bui, Guergana Savova and Lijing Wang assess how random seeds affect fine-tuning of large language models in 2025. Seed effects are often overlooked, yet they can shift results in ways that change how models are compared.
Using two standard language benchmarks, the team measured overall scores such as accuracy across many runs and tracked their spread. They also add a new consistency metric that captures how stable single predictions are from run to run. Large variance appeared at both the overall and the single-prediction level. The authors urge care with seeds in tuning and evaluation.
Catalog record
| CREATOR | Hao Zhou, Guergana Savova, Lijing Wang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07329
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessing the Macro and Micro Effects of Random Seeds on Fine-Tuning Large Language Models Hao Zhou, Guergana Savova, Lijing Wang Abstract The impact of random seeds in fine-tuning large language models (LLMs) has been largely overlooked despite its potential influence on model performance. In this study, we systematically evaluate the effects of random seeds on LLMs using the GLUE and SuperGLUE benchmarks. We analyze the macro-level impact through traditional metrics like accuracy and F1, calculating their mean and variance to quantify performance fluctuations. To capture the micro-level effects, we introduce a novel metric, consistency , measuring the stability of individual predictions across runs. Our experiments reveal significant variance at both macro and micro levels, underscoring the need for careful consideration of random seeds in fine-tuning and evaluation. 1 Introduction The impact of random seeds in neural network training has long been recognized across various domains, such as general machine learning classification and regression tasks Ganesh et al. 2023 ; Madhyastha and Jain 2019 , computer vision Picard 2021 ; Åkesson et al. 2024 , natural language processing (NLP) Bethard 2022 ; Lucic et al. 2022 . Random seeds influence initialization and training dynamics, introducing variability in model outcomes that can lead to significant fluctuations in performance Bengio 2012 . In the field of NLP, large language models (LLMs) have achieved remarkable success across a wide range of NLP tasks, setting new state-of-the-art results on benchmarks like GLUE and SuperGLUE. These benchmarks have became the de facto standard for evaluating the capabilities of LLMs in understanding, reasoning, and generating natural language. Despite the success of LLMs, the pretrained transformer architectures, such as BERT Devlin et al. 2019 and RoBERTa Liu 2019 , have been found to be particularly sensitive to random seeds Risch and Krestel 2020 ; Dodge et al. 2020 ; Mosbach et al. 2021 . This sensitivity can result in substantial performance variations, complicating the interpretation of experimental results and undermining confidence in benchmarking or state-of-the-art outcomes. A recent analysis of 85 papers from the ACL Anthology Bethard 2022 revealed risky practices in the use of random seeds: over 50% of the papers exhibited potential misuse, with 24 using a single fixed random seed. This highlights that the influence of random seeds on LLM performance is still an underexplored area. Figure 1: Macro and micro performance. A pretrained LLM is fine-tuned with random seed 42 and 52. The accuracy for both models is 60%, but the overlapping of individual predictions is 20%. Existing studies examining the impact of random seeds Ganesh et al. 2023 ; Madhyastha and Jain 2019 ; Picard 2021 typically evaluate performance variations by measuring the variance of standard metrics, such as accuracy and F1 score for classification tasks, or Pearson correlation for regression tasks, across multiple seeds. These evaluations focus on the macro-level agreement of model performance across the entire test set, offering insights into overall variability. However, they overlook the micro-level impact of how individual test points are influenced by random seed variations . As shown in Figure 1 , model performance is robust to random seeds 42 and 52 at the macro level (both achieve 60% accuracy) but lacks consistency at the micro level (only 20% overlapping predictions). This micro-level inconsistency can have severe consequences in real-world applications, especially in fields where model predictions are highly sensitive to individual test points, such as medical diagnosis and autonomous driving. Understanding this micro-level effect is crucial for assessing model robustness at the level of individual predictions, ensuring that specific test samples are not inconsistently misclassified or predicted due to seed-induced variations. Additionally, it helps pinpoint specific areas where models may exhibit significant instability, such as consistently misclassifying certain types of data points or showing highly variable predictions for similar inputs. Recognizing these areas of instability can guide targeted improvements in both model design and evaluation practices, ensuring that assessments account for seed-induced variations in performance. Major contributions : To address these gaps, in this work, (1) we analyze the impact of random seeds on pretrained LLMs using the GLUE and SuperGLUE benchmarks, covering both macro and micro-level variability; (2) We introduce a novel consistency metric to assess prediction stability on individual test points, capturing the micro-level effects of random seeds; (3) Our extensive experiments reveal significant variability in both standard and consistency metrics, underscoring the need to consider seed-induced variations in fine-tuning and evaluation, and incorporate random seed sensitivity into benchmarking and reporting for more reliable and reproducible results. 2 Macro Metric: Variance To measure the macro-level impact of random seeds on LLM performance, we calculate the variance of a standard metric across multiple seeds. Let [\zeta_{1},\cdots,\zeta_{S}] represent the values of a model performance metric for LLMs fine-tuned with S random seeds, the variance is calculated by: \text{VAR}(\zeta)=\sqrt{\frac{1}{S}\sum_{i=1}^{S}(\zeta_{i}-\bar{\zeta})^{2}} (1) where \bar{\zeta}=\frac{1}{S}\sum_{i=1}^{S}\zeta_{i} . \zeta can be any standard metrics, such as F1 score for classification tasks or Pearson correlation for regression tasks. A smaller VAR indicates less variation in macro-level performance. 3 Micro Metric: Consistency ’Consistency’ can have varying definitions across domains. Building on prior work, Wang et al. 2020 formally defined the consistency of a deep learning model as its ability to produce consistent predictions for the same input when periodically retrained with streaming data in deployment settings. Extending this idea, we define the consistency of an LLM as its ability to generate consistent predictions for the same input across models fine-tuned with different hyperparameter settings, with correct-consistency further specifying its ability to make consistent correct predictions in this context. More specifically, consider two LLMs A and B , given a dataset \mathcal{D}={d_{1},\cdots,d_{N}} of N data points, y_{i}^{A} and y_{i}^{B} are the prediction of A , B for a data point d_{i} with ground truth r_{i} . For classification tasks , we calculate the consistency as follows: \text{CON:}\frac{1}{N}\sum_{t=1}^{N}{1_{A,B}(t)} (2) where 1_{A,B}(\cdot) is the indicator function that equals 1 if y_{t}^{A}=y_{t}^{B} , otherwise 0. And the correct-consistency is calculated by: \text{ACC-CON:}\frac{1}{N}\sum_{t=1}^{N}{1_{A,B,r}(t)} (3) where 1_{A,B,r}(\cdot) is the indicator function that equals 1 if y_{t}^{A}=y_{t}^{B}=r_{t} , otherwise 0. For regression tasks , we calculate the consistency as the Pearson correlation between two sets of predictions [y_{1}^{A},...,y_{N}^{A}] and [y_{1}^{B},...,y_{N}^{B}] : \text{CON-PEAR:}\frac{\sum_{i=1}^{N}\left(y_{i}^{A}-\overline{y^{A}}\right)\left(y_{i}^{B}-\overline{y^{B}}\right)}{\sqrt{\sum_{i=1}^{N}\left(y_{i}^{A}-\overline{y^{A}}\right)^{2}}\sqrt{\sum_{i=1}^{N}\left(y_{i}^{B}-\overline{y^{B}}\right)^{2}}} (4) CON and ACC-CON range from [0, 1], while CON-PEAR ranges from [-1, 1], where higher values indicate smaller variations in micro-level predictions. While consistency metrics can generally be used for quantifying the agreement of individual predictions from any two LLMs with different architectures, hyperparameters, or training settings, in our study, they are specifically used to serve as metrics to evaluate the micro-level impact of random seeds on the same pretrained LLM. 4 Experimental Setup In this section, we will show the details of our experiment settings. More setting details can be found in Appendix. 4.1 Benchmarks In this study, we conduct experiments on a range of NLP tasks including COLA (Matthews Correlation Coefficient), SST2 (Accuracy), MRPC (Accuracy), STSB (Pearson correlations), QQP (Accuracy), QNLI (Accuracy), and RTE (Accuracy) from GLUE Wang et al. 2018 benchmark; RTE (Accuracy), CB (Accuracy), WiC (Accuracy), BoolQ (Accuracy), MultiRC (Accuracy), and COPA (Accuracy) from SuperGLUE Wang et al. 2019 benchmark. In our paper, we use RTEG to denote RTE task from GLUE and RTES for SuperGLUE. To ensure our experiments are conducted with