Testing the validity of multiple opinion dynamics models Samuel Moor-Smith 1, Dino Carpentras 1 Abstract While opinion dynamics models have been extensively studied as stylized models, there has been growing attention to the possibility of combining these models with empirical data. This attention seems to be driven by the many social issues that strongly depend on people’s opinions (such as climate change and vaccination) and the need for empirically valid models to design related policy interventions. While different models have been combined in various ways with empirical data, standardised comparison of models against empirical data is still lacking. In this article, we test the validity of multiple opinion dynamics models—including both stylized and more realistic models. Our approach follows a “data science-like” validation procedure, where we first calibrate the model’s free parameters using an initial range of years (e.g. 2010-2015), and then use data from one wave (e.g. 2016) to predict data in the following wave (e.g. 2017). We initially tested such a procedure using simulated data and then tested different models on various topics from the European Social Survey. Both toy models and empirical models perform well on the simulated data, but fail to predict future years in the empirical data. Furthermore, during the calibration phase on the empirical data, most models learned to “freeze”—meaning that their predictions for the following year are just a copy of the data from the previous year. This work advances the literature by offering a benchmark for comparing different opinion dynamics models. Furthermore, our tests show that real-world dynamics appear to be completely incompatible with the dynamics of the tested models. This calls for more effort in exploring what are the features that would improve validity and applications for opinion dynamics models. † † institute: ETH Zürich, 8092, Zürich, Switzerland , WWW home page: https://coss.ethz.ch/ 1 Introduction For a long time, opinion dynamics has focused on stylized models that are able to show some interesting properties of opinion interactions ( 20 ; 8 ; 34 ) . For example, models like the Deffuant model ( 10 ) or the Axelrod model ( 1 ) have shown how simple interactions can lead to the formation of coherent opinion-based groups, similarly to how the Schelling model has shown the appearance of segregation patterns ( 29 ) . These models have driven the field of opinion dynamics and provided many valid interpretations of how opinions form, shift, and evolve based on theories in psychology and sociology. However, in recent years, societies have started facing multiple pressing problems related to public opinion. For example, climate change can be effectively addressed only if people agree on what to do ( 26 ) , vaccines can produce herd immunity only if adopted by most people ( 28 ) , and countries can face instability if people’s ideological divide becomes too strong ( 17 ) . Many of these challenges call for well-informed public policy, which opinion-dynamics models can help to support ( 14 ) . This has driven further research into understanding the more realistic features in stylized models ( 21 ; 23 ) . Others have attempted a more empirical approach by deriving models from experiments, or testing if their models could reproduce some empirical patterns ( 22 ; 2 ; 7 ; 11 ; 27 ) . While all these approaches explore some kind of model validity, they present at least one of the following three main limitations: (1) data and methods are often very specific and do not generalize to other common datasets; (2) it is often not clear if the model is valid or not; and (3) temporal predictions are often neglected. The first problem refers to the fact that most studies rely on data and methodologies that are very context- or model-specific, and make it hard to use a similar methodology in other studies. For example, ( 5 ) used opinion data related to vaccination in 2018 to predict vaccination coverage in 2019. While this is an interesting implementation, it is unclear how this could be used for other datasets and years. In another example, ( 2 ) et al. used macro-level data to estimate a micro-level parameter, a methodology that requires a very specific data collection process. The second problem refers to the fact that results are often given as pure numbers. For example, we may estimate that a model, M_{1} , produces an estimation error of 5.0, while model M_{2} produces an error of 4.8. While everybody can agree that 4.8 is an improvement on 5.0, it is not clear if this improvement is significant in any way. For example, simply presenting the results in this way, it may seem as though M_{2} performs well on empirical data. However, we may discover that a purely random model produces an error of 2.3, thus even if M_{2} is an improvement to M_{1} it still performs so poorly that it should not be used for any empirical application. The third and final problem relates to the predictive ability of these models. There is a general split in the social simulation community regarding whether the purpose of such models is primarily explanatory or predictive ( 31 ) . In this paper, we adopt an empirical perspective and therefore consider predictive performance as a key criterion for assessing model validity ( 16 ; 9 ; 12 ) . This does not mean that the model must be deterministic, as we can still have valid and invalid models for probabilistic processes. For example, for a fair dice, a valid model would be one that predicts 1 to 6 with equal probability, while supposing 1 as the unique possible result would be an invalid model. Similarly, an opinion dynamics model that often makes bad predictions 1 1 1 By “prediction” here we mean the ability of a model to estimate data that are in the future respect to the input data. For example, using 2010 data to “predict” 2015 data. Thus, our use of prediction is only related to the temporal positioning of the data and has nothing to do with “the future” with respect to the time of writing. should not be used for policy applications. Despite this key role of prediction, they are rarely used for testing opinion dynamics models’ validity. For all these reasons, the remainder of the article presents a methodology for testing multiple opinion dynamics models. Specifically, Section 2 outlines our approach to model calibration and validation; Section 3 evaluates each model’s performance in predicting and reconstructing synthetic opinion data; and Section 4 applies the models to empirical data. 2 Methodology for Model Calibration and Validation In this section, we describe the terminology and methods used to calibrate and validate the opinion dynamics models, including the loss function, the explained variance metric, the optimization algorithm, and the general evaluation framework applied across all experiments. 2.1 Terminology and notation 2.1.1 Datasets: We refer to a dataset D as a collection of opinion distributions O_{t} , each one related to a time t . For example, a dataset containing opinion distributions annually collected from 2002 to 2018 will be represented as D=\{O_{t}\}_{t=2002}^{2018} . 2.1.2 Models as operators: For our purposes, an opinion dynamics model M is an operator that takes in an opinion distribution O_{t} as an input and produces a new opinion distribution \hat{O}_{t+1} , where t represents a wave of the dataset 2 2 2 Note that consecutive waves may not be in consecutive years. For example, data may be collected every 2 years, and so the wave after 2016 will be 2018. . For example, if we are applying model M on 2016 data to predict 2018 data, we will write M(O_{2016})=\hat{O}_{2018} . This notation will become useful when we will compare the “original” 2018 data (i.e. O_{2018} ) with the one predicted from the model (i.e. \hat{O}_{2018} ). Furthermore, we will refer to the original data as the ground-truth 3 3 3 While most empirical datasets on opinions, attitudes, or values are cross-sectional, meaning that different individuals are surveyed at each time point, we assume that with a sufficiently large sample size the aggregate (macro-level) characteristics remain representative of the underlying population. The models are designed to capture population-level dynamics, so this assumption allows us to use the available cross-sectional data as a practical—albeit imperfect—proxy for longitudinal ground-truth. . This term will be used both when the original data are empirical and when they have been obtained from other simulations. 2.1.3 Prediction Error: Given a certain model M we define its prediction error for a certain dataset (consisting of T opinion distributions) as: E(M,D)=\sum_{t=2}^{T}\Delta(O_{t},\hat{O}_{t}) (1) where \Delta is the Wasserstein distance between the two distributions ( 32 ) . The Wasserstein metric implementation we use is from scipy ( 33 ) . \hat{O}_{t} is the models’ predicted value for timestep, t : \hat{O}_{t}=M(O_{t-1}) . 2.1.4 Explained Variance and Opinion Drift: While the error can tell us which models perform better or worse, it cannot directly tell us if a model is performing well or not. Indeed, we know that an error of 1.1 is better than an error of 1.2, but we cannot really tell if this is an excellent or very poor result. To avoid this problem, we introduce a null (“No-Change”) model. Such a model, called M_{0} , predicts that the following distribution will simply be identical to the previous distribution, so that: M_{0}(O_{t})=O_{t} (2) Thus the prediction error of the null model will always be equal to the variance in the data between all of the years: E(M_{0},D)=\sum_{t=2}^{T}\Delta(O_{t},O_{t-1}) (3) Since E(M_{0},D) relies solely on the dataset and also captures the amount of opinion change from one timestep to the next, we can define a new term: \Delta_{\text{op}}(D)=E(M_{0},D)=\sum_{t=2}^{T}\Delta(O_{t},O_{t-1}) (4) We will call \Delta_{\text{op}}(D) the opinion drift of dataset D and this will be mentioned extensively in this study. Note that a low \Delta_{\text{op}}(D) implies that there is little opinion change in the dataset across timesteps, and a high \Delta_{\text{op}}(D) implies that there is a large amount of opinion change across timesteps. We can now use this null model to produce a normalized measurement of a model M ’s prediction using the formula of the explained variance V: V(M,D)=1-\frac{E(M,D)}{E(M_{0},D)}=1-\frac{E(M,D)}{\Delta_{\text{op}}(D)} (5) Similarly to the classical explained variance, when model a M ’s predictions are perfect (i.e. E(M,D)=0 ) the explained variance is 1; when the model performs as well as the null model (i.e. E(M,D)=E(M_{0},D) ), the explained variance is 0; and if our model performs worse than the null model, the explained variance is negative. 4 4 4 The introduction of the opinion drift and, consequently, the normalized explained variance enables us to compare the performance of variance models in a standard way. The null model is an unbiased benchmark to compare the provided model against. 2.2 Models and Free Parameters We conduct our evaluation on four opinion dynamics models, comprising two foundational “toy” models and two empirically motivated models. The toy models include the Deffuant Model ( 10 ) and the Hegselmann–Krause (HK) Model ( 25 ) , both based on the principle of bounded confidence. The empirical models include the ED (Experimentally-Derived) Model ( 6 ) , whose dynamics were derived from experimental data, and the Duggins Model ( 13 ) , which incorporates multiple psychological effects through mechanisms such as intolerance, susceptibility and conformity. Each opinion dynamics model includes a set of free parameters that can be tuned to control the model’s behaviour and the dynamics of opinion evolution. For example, in the Deffuant model ( 10 ) , the parameters include \mu (the convergence parameter) and \epsilon (the bounded confidence threshold). When referring to a specific instance of a model with a given parameter set, p , we denote it as M_{p} . 2.3 Optimization Method Assume that we have a ground-truth dataset, D_{\text{true}} , which contains the observed opinion distributions at each time step: D_{\text{true}}=\{O_{1},O_{2},...,O_{T}\} . For the free parameter optimization, we only use a subset of D_{\text{true}} , which we denote by D_{\text{opt}}\subset D_{\text{true}} . This subset consists of the first T_{\text{opt}} timesteps of D_{\text{true}} : D_{\text{opt}}=\{O_{1},O_{2},\ldots,O_{T_{\text{opt}}}\},\quad\text{ where }T_{\text{opt}}0. This threshold varies by model: the HK model requires a negligible amount of opinion change to surpass the baseline, whereas the ED model requires a significantly larger change. We find that the amount of change necessary to outperform the baseline is linked with the stochasticity of the model. The HK and Deffuant models—particularly the HK—are relatively deterministic compared to the two more empirical models. As a result, the amount of change needed to identify model effects is relatively low. Figure 1 : Explained variance of the optimized model (coloured) and generator model (grey) as a function of \Delta_{\text{op}}(D) . For the (grey) ground-truth case (refer. Sec. 3.1 ), deterministic models like Deffuant and HK require little opinion drift to outperform the null baseline, while more stochastic models like ED and Duggins require substantially more. The Duggins model shows high variance in repeated runs, reflecting its limited reproducibility. Furthermore, The optimizer matches ground-truth performance at high \Delta_{\text{op}}(D) and often exceeds it at low \Delta_{\text{op}}(D) by mimicking the null model. Shaded areas indicate standard deviation across runs. 3.1.2 A Note on the Duggins Model and Reproducibility The Duggins model ( 13 ) stood out in our evaluation: even with identical parameters, 72% of runs performed worse than the naive null model. Two factors explain this: (1) The parameter ranges used here 5 5 5 Values: \mu_{t}=(0.7,1.0),\mu_{s}=(1.0,5.0),\mu_{c}=(0.1,0.5) , with \sigma_{t},\sigma_{c}=0.3 , \sigma_{s}=0.7 . Grid size: 1000. Social radius: \mu_{r}=(0,30) , \sigma_{r}=4 . match those in ( 13 ) , which were selected to preserve strong diversity. With these parameters, the Duggins model produces datasets with very small amounts of opinion drift. While extending the possible parameter values (e.g., by increasing conformity or intolerance) could increase \Delta_{\text{op}}(D_{\text{true}}) —and thereby reduce the influence of stochasticity, making the model appear more reproducible—doing so would compromise the integrity of the model. (2) Unlike the other models, the Duggins model’s “free parameters” are simply the means and standard deviations of distributions from which each agent’s traits are independently sampled. Combined with the spatial structure of the model, this stochastic resampling introduces significant variability: even when the same parameter values (and agent coordinates) are used, each run resamples each agents parameters, leading to substantially different opinion dynamics. As a result, the model is inherently not reproducible (see the grey plot for the Duggins model in Fig. 1 ). 3.1.3 Results of experiment 1 These results reveal that, prior to any optimization, the naive null model, M_{0} , can outperform the true generator model, M_{p^{*}} , when the amount of opinion drift is small (i.e. when the dataset presents little variability). A concrete example of this is a scenario is when we have only one pairwise agent interaction between O_{t} and O_{t+1} : the null model will differ only on that one pair, while the generator model—due to stochastic sampling—may not select the correct interaction and will therefore differ both on the true pair and the one it mistakenly simulates. In such cases, the randomness inherent in the generator model can actually worsen its performance relative to a naive baseline. We also observe that the variance across runs of the generator model differs significantly between models. This reflects the inherent noise in each model’s design and the extent to which results from this model can be reproduced. The ED and Duggins models, in particular, exhibit far greater noise than the Deffuant and HK models. Accordingly, a higher level of opinion change is required before a rerun of the generator model performs better than the null model. These observations are essential for interpreting the results of subsequent experiments. They reinforce the importance of the reproducibility runs, which establish a realistic upper bound on optimizer performance. Since they use the generator model directly, they define the best possible outcome that any optimizer could hope to match under ideal conditions. 3.2 Experiment 2: Validating the Optimizer Having established an upper bound on model performance using known parameters, we now must validate our optimizer. The central question here is whether the optimizer can recover parameters that will result in similar model dynamics. To do this, we run the optimizer on a subset (i.e. D_{\text{opt}}\subset D_{\text{true}} ) of the same ground-truth datasets that were generated in Experiment 1 6 6 6 Recall: if we have D_{\text{true}}=\{O_{1},O_{2},...,O_{T}\} , we will define D_{\text{opt}}=\{O_{1},O_{2},\ldots,T_{\text{opt}}\} , with T_{\text{opt}}