Evolutionary Prediction Games Eden Saig Technion – Israel Institute of Technology Haifa, Israel edens@cs.technion.ac.il, Nir Rosenfeld Technion – Israel Institute of Technology Haifa, Israel nirr@cs.technion.ac.il Abstract When a prediction algorithm serves a collection of users, disparities in prediction quality are likely to emerge. If users respond to accurate predictions by increasing engagement, inviting friends, or adopting trends, repeated learning creates a feedback loop that shapes both the model and the population of its users. In this work, we introduce evolutionary prediction games , a framework grounded in evolutionary game theory which models such feedback loops as natural-selection processes among groups of users. Our theoretical analysis reveals a gap between idealized and real-world learning settings: In idealized settings with unlimited data and computational power, repeated learning creates competition and promotes competitive exclusion across a broad class of behavioral dynamics. However, under realistic constraints such as finite data, limited compute, or risk of overfitting, we show that stable coexistence and mutualistic symbiosis between groups becomes possible. We analyze these possibilities in terms of their stability and feasibility, present mechanisms that can sustain their existence, and empirically demonstrate our findings. 1 Introduction Accurate predictions have become essential for any platform that supports user decision-making. Improvements in prediction accuracy often directly translate to better quality of service with benefits to both the platform and its users. This is one reason why modern platforms ranging from content recommendation and online marketplaces to personalized education and medical services have come to rely on machine learning as their backbone, and are investing much effort and resources in continually improving their predictions. However, while generally beneficial, promoting accuracy blindly can have unexpected, and in some cases undesired, consequences [ 9 , 59 , 24 ] . It is therefore important to understand the possible downstream and long-term effects of learning on social outcomes. Conventional learning approaches aim to maximize accuracy on a given, predetermined data distribution. But in social settings, the distributions are composed of those users who choose to use the platform. When such choices depend on the quality of predictions, learning becomes a driver of user self-selection, and thus gains influence over its user population. This creates a feedback loop: model deployment shifts the population, and population changes trigger model retraining. We are interested in understanding the general tendencies and possible long-term outcomes of this process. In particular, our focus is on user feedback dynamics in which accurate predictions encourage engagement or adoption, and prediction errors discourage them. Such dynamics arise across domains: accurate recommendations drive network-effect growth as users invite peers [ 39 ] ; successful content strategies on social media are mimicked by others [ 4 ] ; precise credit-risk models reduce premiums and therefore attract more clients to loan programs [ 32 ] ; and medical providers with higher diagnostic accuracy draw more patients [ 3 ] . Common to the above is that individual choices adhere to some form of group structure in the population, and that in aggregate, these groups tend to grow when prediction quality is higher. Note that this notion of “group” is flexible, as groups can represent different demographics, behaviors, or roles, and with memberships being either inherent or chosen. Focusing on user choices as the primary driving force, we seek to analyze the impact of learning on group proportions over time. For this purpose, we adopt a novel evolutionary perspective and model the joint dynamics of learning and user choices through the lens of natural selection . Under the assertion that some degree of predictive error always exists, our key modeling point is that accuracy becomes, in effect, a scarce resource over which different groups in the population “compete”. By associating each group’s accuracy with its evolutionary fitness, we obtain an evolutionary process in which learning is the driver of selective pressure, and consequently, a determinant of long-term population outcomes. We can then ask questions regarding the long-term tendencies of the population composition, overall and per-group accuracies, and the affect of different modeling and algorithmic choices on temporal trends and long-term evolutionary outcomes. Contributions. Our main conceptual innovation is to analyze these feedback loops using a population game —a core component of evolutionary game theory useful for studying the dynamics of agents driven by local interaction rules [ 41 , 27 , 52 ] . Evolutionary game theory relates between game-theoretic properties of population games (e.g., Nash equilibria), and the dynamics of a wide variety of natural selection dynamics, such as imitation, word-of-mouth influence, or social learning, all of which are supported by our analysis. This makes the framework well-suited for our setting, as it allows us to address the common thread in the wide variety of feedback loops described above in a unified way. Towards this, we first propose a novel game, called an evolutionary prediction game , in which evolutionary fitness is associated with prediction accuracy. This allows us to study the co-evolution of a (re)trained model and the population of its users. We then analyze the structure of games induced by different learning algorithms and their corresponding dynamics, and characterize the possible outcomes under different settings. Our results include conditions under which ‘survival of the fittest’ is a likely outcome; mechanisms that nonetheless enable coexistence; a discussion of the role and effects of retraining; and connections between survival, accuracy, and (evolutionary) stability. We also discuss connections to fairness and social conservation. We conclude by complementing our analysis with experiments using both synthetic and real data and coupled with simulated dynamics. Our empirical results shed light on when and how certain user groups are likely to dominate, disappear, or coexist, and demonstrate how different design choices can shape social outcomes—even if inadvertently. Together, these highlight the importance of understanding, anticipating, and accounting for the long-term effects of learning on its user population. 1.1 Related Work Evolutionary game theory. Evolutionary game theory was originally developed as a framework for modeling natural selection processes in biology [ 44 , 42 , 41 , 27 , 56 ] , and has since been applied to the analysis of large populations of myopic interacting agents in economics [ 52 , 20 ] , and to the study of online learning algorithms [ 19 , 7 ] . From a technical standpoint, the games we propose and analyze are unique in this space as they are defined implicitly as solutions to statistical optimization problems. Our results for oracle classifiers reflect the principle of competitive exclusion [ 23 ] , while our analysis of non-optimal classifiers relates to long-studied questions of coexistence [ 11 ] . Performativity. Our work relates to the emerging field of performative prediction [ 48 ] , which studies learning in settings where model deployments affect the underlying data distribution, with emphasis on (re)training dynamics and equilibrium. A central effort in this area is to identify general global properties (e.g., appropriate notions of smoothness) that guarantee convergence. Our work complements this effort by proposing structure , in the form of user self-selection [ 65 , 28 ] , which provides an efficient low-dimensional representation of the distribution map. This allows us to work in the notoriously challenging stateful performative setting [ 5 ] , and to characterize stronger notions of stability. Fairness. The notion of equilibrium in evolutionary prediction games relates to the fairness criterion of overall accuracy equality [ 61 ] . The main distinction is that in prediction games, the question is not only whether groups are treated fairly, but also which groups are even considered, since fairness can be measured only for groups that are observed. Complementing other voiced concerns on long-term fairness outcomes [ 25 , 40 , 14 , 49 ] , our work adds a novel counterfactual perspective: if we observe fair outcomes at present, could this be because some groups were historically driven out of the game? See Appendix B for an extended discussion and additional related literature. Figure 1: Natural selection in a two-group setting. Population is a mixture {\bm{p}} of groups; Classifier h is learned using data from the mixture distribution D_{\bm{p}} ; Evolutionary fitness is associated with prediction accuracy; Differences in fitness drive change in mixture coefficients; Possible long-term tendencies are dominance (only A survives), extinction (only B survives), or coexistence (both survive). 2 Setting The core of our setting is based on a standard supervised learning setup in which examples describe user data. Denote features by x\in\mathcal{X} and labels y\in\mathcal{Y} . For a given distribution D over pairs (x,y) , and given a training set S={\left\{{(x_{i},y_{i})}\right\}}_{i=1}^{n}\sim D^{n} , the goal in learning is to use S to find a predictor h:\mathcal{X}\to\mathcal{Y} from a class \mathcal{H} which minimizes the expected error under some loss function \ell:\mathcal{Y}\times\mathcal{Y}\to\mathbb{R} . For concreteness, we focus on classification tasks with the 0-1 loss, \ell(\hat{y},y)=\mathds{1}\left\{{\hat{y}\neq y}\right\} , whose minimization is equivalent to maximizing expected accuracy: \mathrm{acc}_{D}(h)=\mathbb{P}_{(x,y)\sim D}{\left[{h(x)=y}\right]} (1) Denote the learning algorithm by \mathcal{A}(S) , and by h\sim\mathcal{A}(D) the process mapping distributions to learned classifiers, with randomness due to sampling. Since solving Eq. ( 1 ) is both computationally and statistically hard, practical learning algorithms often resolve to optimizing an empirical surrogate on the sampled data. One question we will ask is how such compromises affect long-term outcomes. Population structure. As data points (x,y) are generated by users, the data distribution D represents a population. We assume the population is a mixture of K groups. Each group k\in[K] is associated with a group-specific distribution D_{k} over (x,y) pairs, which remains fixed, and with a relative size p_{k} , which evolves over time. The overall data distribution is a mixture denoted by D=D_{\bm{p}}=\sum_{k}p_{k}D_{k} , where p_{k} is the current relative proportion of group k (such that \sum_{k}p_{k}=1 and p_{k}\geq 0\,\,\forall k ). Given some h , the marginal expected accuracy on group k is \mathrm{acc}_{k}(h)=\mathrm{acc}_{D_{k}}(h) . The population state vector {\bm{p}}=(p_{1},\dots,p_{K}) will be our main object of interest. As {\bm{p}} encodes group proportions, the space of population states is the (K-1) -dimensional simplex \Delta^{K} , and since {\bm{p}} determines D_{\bm{p}} , for brevity we denote h\sim\mathcal{A}({\bm{p}})=\mathcal{A}(D_{\bm{p}}) and \mathrm{acc}_{\bm{p}}(h)=\mathrm{acc}_{D_{\bm{p}}}(h) . Dynamics. When a classifier is deployed, users make decisions according to the quality of the predictions they receive. Collectively, individual responses reshape the population composition {\bm{p}} , resulting in a subpopulation shift [ 62 ] . The updated population then serves as the training distribution for the next classifier—which forms a feedback loop between {\bm{p}} and h . For training, we consider a simple procedure of repeated training on the current distribution. In terms of user behavior, our framework accommodates a wide family of dynamics, including reproduction, network effects, imitation, social influence, rational decision-making, and competition, as alluded to in Section 1 . In Appendix C we define these formally and provide additional examples. In Section 3 we make our assumptions explicit. Outcomes. Denote the initial population state by {\bm{p}}^{0} , and the initial classifier by h^{0}\sim\mathcal{A}({\bm{p}}^{0}) . We consider dynamics which evolve towards fixed points, and denote the corresponding point by {\bm{p}}^{*} . Denote the classifier by h^{*}\sim\mathcal{A}({\bm{p}}^{*}) . We characterize fixed points in terms of three key properties: • Population composition: For the population, dynamics may act as a force diminishing certain groups (i.e., driving towards a state satisfying p^{*}_{k}=0 for some k\in[K] ). In such cases, we say that the population is dominated by the remaining groups. When the population composition at rest has at least two groups (i.e., p^{*}_{k}>0 for multiple values of k ), we say that these groups coexist . • Accuracy: For the classifier, dynamics may result in overall performance improving over time, (i.e., \mathrm{acc}(h^{*})>\mathrm{acc}(h^{0}) ), deteriorating, or remaining unchanged. • Stability: For the joint system, some fixed points {\bm{p}}^{*} are stable and attract all neighboring states, while other fixed points are unstable and may cause dynamics to diverge under small perturbations. An illustration of the setting is provided in Figure 1 . 3 Evolutionary Prediction Games Figure 2: Evolutionary dynamics for two groups, induced by an oracle linear classifier ( Section 4 ). (Left) Data distributions in feature space. Dashed line demonstrates the optimal linear classifier h_{\bm{p}} for the uniform mixture {\bm{p}}=(0.5,0.5) . (Center) Evolutionary prediction game F_{k}({\bm{p}})=\mathrm{acc}_{k}(h_{\bm{p}}) . The game has two stable equilibria with single-group dominance, and one unstable coexistence equilibrium. Overall population accuracy \mathrm{acc}_{\bm{p}}(h_{\bm{p}}) is convex, and maximized at the boundries. (Right) Replicator dynamics induced by the game, for various initial states {\bm{p}}^{0} . Populations evolve towards fixed points with single-group dominance. To analyze the feedback loop, we leverage evolutionary game theory , which is a general framework for the analysis of natural selection processes [ 42 ] . The framework models evolutionary interactions between individuals from K groups in a large mixed population: Given a population state {\bm{p}}\in\Delta^{K} , each group k\in[K] is associated with a scalar fitness function , F_{k}({\bm{p}}) , which quantifies the average evolutionary fitness of individuals in that group. Together, the fitness functions F=\left(F_{1},\ldots,F_{K}\right) define a population game F:\Delta^{K}\to\mathbb{R}^{K} , which governs the dynamics of {\bm{p}} . Formally, population games are symmetric normal-form games for a continuum of agents. Evolutionary game theory uses population games to map myopic, often local agent decisions to group-level dynamics governing the evolution of group proportions {\bm{p}} over time. Modeling the game at the group level serves as useful abstraction for studying the general tendencies and possible long-term outcomes of local interactions in aggregate. Note that the notion of “group” is flexible and can support memberships that are either inherent (e.g., by demographics) or chosen (e.g., service provider). Accuracy as evolutionary fitness. We formalize the interplay between learning and population dynamics through a novel type of population game—an evolutionary prediction game . The game associates the evolutionary fitness of each group with its expected marginal accuracy \mathrm{acc}_{k} under a classifier h trained on data sampled from the mixture D_{\bm{p}} : Definition 1 (Evolutionary Prediction Game) . Let \mathcal{A} be a learning algorithm, and let D_{1},\dots,D_{K}\in{\Delta\left({{\mathcal{X}\times\mathcal{Y}}}\right)} be group distributions. Denote by \mathrm{acc}_{k}(h) the marginal accuracy of group k\in[K] under classifier h\sim\mathcal{A}({\bm{p}}) . The evolutionary fitness of group k under population state {\bm{p}} is: F_{k}({\bm{p}})=\mathbb{E}_{h\sim\mathcal{A}({\bm{p}})}{\left[{\mathrm{acc}_{k}(h)}\right]} (2) Together, the tuple F({\bm{p}})=(F_{1}({\bm{p}}),\dots,F_{K}({\bm{p}})) defines an evolutionary prediction game . In the notation F_{k}({\bm{p}}) , we assume for brevity that the learning algorithm \mathcal{A} and group distributions D_{k} are clear from context, and we state them explicitly otherwise. Evolutionary prediction games capture an implicit interaction between groups: while each group’s fitness F_{k}({\bm{p}}) is given by the accuracy of its own members, the classifier h_{\bm{p}} is trained using data from the entire population. Thus, the evolution of different groups becomes coupled through the learning process. For example, Figure 2 (Left) illustrates a binary classification setting over two groups {\left\{{A,B}\right\}} . Figure 2 (Center) shows the corresponding evolutionary prediction game F({\bm{p}}) induced by an oracle linear classifier (defined in Section 4 ). At {\bm{p}}=(0.7,0.3) , we have F_{A}({\bm{p}})\approx 0.97 and F_{B}({\bm{p}})\approx 0.78 , and thus the fitness of group A is higher when the classifier is trained on data sampled from D_{{\bm{p}}} . Equilibrium. Nash equilibrium in general population games is defined as follows: Definition 2 (Nash equilibrium of a population game; e.g. [ 52 ] ) . Let F({\bm{p}}) be a population game, and denote \support({\bm{p}})={\left\{{k\mid p_{k}>0}\right\}} . A state {\bm{p}}^{*}\in\Delta^{K} is a Nash equilibrium of F({\bm{p}}) if it satisfies: \support({\bm{p}}^{*})\subseteq\argmax\nolimits_{k\in[K]}F_{k}({\bm{p}}^{*}) (3) This also applies to evolutionary prediction games; e.g., the game in Figure 2 (Center) has three Nash equilibria at {\left\{{(1,0),(0.5,0.5),(0,1)}\right\}} , and all population games have at least one equilibrium [ 52 ] . Induced population dynamics. Formally, we encode dynamics as \dot{\bm{p}}=V_{F}({\bm{p}}) , where \dot{\bm{p}} is the time derivative of {\bm{p}} , and V_{F}:\Delta^{K}\to T\Delta^{K} is a tangent vector field induced by the game F({\bm{p}}) . Our results apply under mild structural assumptions on V_{F} , namely: (i) V_{F} is continuous; (ii) the direction of flow is aligned with fitness, formally V_{F}({\bm{p}})\cdot F({\bm{p}})>0 whenever V_{F}\neq 0 , a property known as positive correlation ; (iii) Fixed points of the dynamical system coincide with equilibria of F({\bm{p}}) , formally through either Nash stationarity or imitative dynamics . All are satisfied by the user-level dynamics highlighted in Section 1 and Appendix C , where they emerge as group-level properties in the large-population limit. Moreover, many of them converge towards the canonical replicator equation , \dot{p}_{k}=p_{k}\left(F_{k}({\bm{p}})-\bar{F}({\bm{p}})\right) , where \bar{F}({\bm{p}})=\sum_{k^{\prime}}p_{k^{\prime}}F_{k^{\prime}} is the average fitness across the population [ 57 ] . We also note that time-scales of convergence are determined (and can be adjusted) by the scale of V_{F} . See Section D.2 for formal definitions. Fairness. In the context of fairness, a classifier satisfies overall accuracy equality if all groups receive equal prediction accuracy [ 61 ] . We show that this criterion is satisfied in expectation by classifiers trained on equilibria mixtures of evolutionary prediction games (proof in Section D.3.2 ): Proposition 1 . Let F({\bm{p}}) be an evolutionary prediction game induced by a learning algorithm \mathcal{A} , and let {\bm{p}}^{*} be a Nash equilibrium. Then h\sim\mathcal{A}(D_{{\bm{p}}^{*}}) satisfies overall accuracy equality in expectation. As outcomes of the dynamics correspond to equilibria of a game, this lends to a natural form of emergent fairness. However, note that this criterion only applies to groups that appear in the data: in Section 6.3 , we further show that a system may appear as fair due to other groups being driven out. Heterogeneous fitness. Finally, in some settings, incentives or outside alternatives may vary across groups. This may be captured using the notion of retention functions [ 25 ] , which translate marginal prediction accuracy to the tendency to remain engaged. In such cases, the transformed fitness function becomes F_{k}({\bm{p}})=\nu_{k}\left(\mathbb{E}_{h_{\bm{p}}}{\left[{\mathrm{acc}_{k}(h)}\right]}\right) , where \nu_{k}:[0,1]\to\mathbb{R} is a group-dependent, continuous strictly-increasing function. A simple example is \nu_{k}(x)=x-b_{k} (for b_{k}>0 ) which allows groups to differ in the minimum accuracy their users require to keep using the system. All of our results apply to any set of strictly monotone \nu_{k} for K=2 , and to positive affine \nu_{k} with per-group offsets b_{k} for K>2 . For clarity we state our main results for \nu_{k}(x)=x , but also discuss them for broader \nu . 4 Competitive Exclusion Under Oracle Classifiers To understand how populations and classifiers evolve jointly, we start by characterizing natural selection in an ‘ideal’ setting, in which a classifier is trained repeatedly using unlimited resources: Definition 3 (Oracle classifier) . Let \mathcal{H} be a hypothesis class, and let D be a data distribution. An oracle classifier with respect to \mathcal{H} is a minimizer of the 0-1 loss in expectation over D : h^{\mathrm{opt}}\in\argmin\nolimits_{h\in\mathcal{H}}\mathbb{E}_{(x,y)\sim D}{\left[{\mathds{1}\left\{{h(x)\neq y}\right\}}\right]} (4) Oracle classifiers represent ERM classifiers in the population limit. This regime abstracts away the complexities due to estimation and approximation errors of practical learning algorithms (i.e., algorithms that learn from finite data in reasonable time). Nonetheless, analysis of the dynamics remains challenging due to the \mathrm{argmin} operator. We focus on settings in which h^{\mathrm{opt}} exists, and denote by \mathcal{A}^{\mathrm{opt}}(D) the (theoretical) learning algorithm which returns the oracle classifier with respect to D . 1 1 1 The Bayes-optimal classifier h^{\mathrm{Bayes}}(x)=\argmax_{y}p(y\,|\,x) is a special case of this definition for \mathcal{H}=\mathcal{Y}^{\mathcal{X}} . We assume that fitness functions are continuous, and tie-breaking is consistent. To simplify presentation, we first assume that each group has a distinct oracle classifier, and then address the general case. Our central result characterizes natural selection induced by oracle classifier retraining: Theorem 1 . Let \mathcal{H} be a hypothesis class, and denote by \mathcal{A}^{\mathrm{opt}} the oracle learning algorithm with respect to \mathcal{H} . Assume that at each population state {\bm{p}} the oracle classifier h_{\bm{p}}\sim\mathcal{A}^{\mathrm{opt}}(D_{\bm{p}}) is deployed. Then it holds that: 1. Accuracy: Overall accuracy increases over time, \tfrac{\mathrm{d}}{\mathrm{d}t}\mathrm{acc}_{{\bm{p}}}(h_{{\bm{p}}})\geq 0 . 2. Stability: A stable equilibrium always exists, and there can be multiple such equilibria. 3. Competitive exclusion: For all stable equilibria, {\left|{\support({\bm{p}}^{*})}\right|}=1 . 4. Coexistence: Equilibria with {\left|{\support({\bm{p}}^{*})}\right|}\geq 2 may exist, but are unstable. Proof sketch. The proof leverages structural properties that emerge despite the complexity of the learning problem. Our main technical tool is the framework of potential games , which are population games that can be expressed as a gradient of a scalar potential function defined over the simplex. We start by showing that the expected accuracy of any fixed classifier is linear in {\bm{p}} , then the core of the proof is a convexity argument: the optimality of \mathcal{A}^{\mathrm{opt}} implies that \mathrm{acc}_{\bm{p}}(h_{\bm{p}}) is convex as a pointwise maximum of linear functions, with gradients given by marginal accuracies. From this we identify f({\bm{p}})=\mathrm{acc}_{\bm{p}}(h_{\bm{p}}) as a potential function, and leverage the known correspondence between equilibria of population games and local extrema of potential functions. Finally, distinct oracle classifiers imply strict convexity of f({\bm{p}}) , and the stability of single-population states follows from the fact that convex functions over the simplex are locally-maximized at the vertices, and the accuracy condition follows from the fact that f({\bm{p}}) is a Lyapunov function. Full proof appears in Section D.5 . Extensions. To simplify presentation, the statement of Theorem 1 assumes that the oracle classifiers for each group are distinct. When this assumption doesn’t hold (e.g., when D_{k}=D_{k^{\prime}} for some k\neq k^{\prime} ), a generalized version of the theorem applies: rather than single-group stable equilibria, dynamics converge towards convex combinations of groups that share an identical oracle classifier with identical marginal accuracy (See Section D.5.1 ). For heterogeneous fitness where retention functions \nu_{k} may differ across groups, we show that the retention functions we support induce potential games with transformed potential functions. Survival is determined according to the maximizers of the transformed potential function, and the quantity that increases over time becomes the average fitness \sum_{k}p_{k}\nu_{k}\left(\mathrm{acc}_{k}(h_{\bm{p}})\right) (See Section D.5.2 ). Finally, we also note that a similar characterization also holds for classifiers trained once at the outset (See Section D.4 ). Interpretation. Theorem 1 suggests that for repeatedly trained oracle classifiers, natural selection will amplify the dominance of groups having the highest fitness at the outset. From an evolutionary perspective, this reflects the competitive exclusion principle [ 23 ] , which states that natural selection tends towards exclusive survival of the fittest when multiple species compete over the same resources. Stability implies that once a group dominates, other groups will remain excluded. While this is the general trend, coexistence is nonetheless enabled by retraining, in the form of possible equilibria in which multiple groups survive. That is, there may exist population states {\bm{p}}^{*} such that p^{*}_{k}>0 for several groups k , and for which this remains to hold when the classifier is retrained. The crux, unfortunately, is that such points are evolutionarily unstable , in the sense that small perturbations around {\bm{p}}^{*} will push dynamics away. This implies that optimal retraining—unconstrained by limited data and compute—induces a strong form of competition between groups. But the existence of mixed equilibria suggests that other algorithms might be able to encourage symbiosis, which we explore next. 5 Avenues for Coexistence Figure 3: Graphical illustration of theoretical coexistence results. Top row presents data distributions, bottom row shows the corresponding prediction games. (Left) Stable and mutualistic coexistence induced by the use of a proxy loss ( Theorem 2 ). (Center) Stable coexistence induced by interpolation ( Section D.7 ). (Right) Mutualistic coexistence induced by a finite training set ( Section D.8 ). Intuitively, oracle classifiers drive dynamics away from mixture equilibria because the learning objective is fully aligned with, and so reinforces, self-selection. This motivates the natural question: Can repeated training drive dynamics also towards coexistence, and when? Here we illustrate two ways for symbiosis to arise: as a result of inherent limitations of learning in practice (i.e., limited data or compute), and by applying an ad-hoc, dynamics-aware learning algorithm designed to stabilize outcomes. A context-dependent view on coexistence. It is important to emphasize that coexistence in our setting is neither innately ‘good’ or ‘bad’, but rather, its desirability depends on context. Hence, the question of whether coexistence should be encouraged depends on the task at hand, the nature of user choices, and the goals of the system designer. For instance, recommendation platforms may aim to preserve both content and audience diversity, making coexistence desirable. Conversely, in consumer markets having multiple competing standards (e.g., video encoding format, compression schemes, payment platforms), there are benefits to converging on the single best alternative. This context-dependent stance aligns well with the notion of evolution, which in itself is neutral. 5.1 Learning in Practice While the goal in learning is to minimize the expected 0-1 loss, in practice learning often resorts to optimizing proxy objectives over sampled data via empirical risk minimization. Here we show that, perhaps surprisingly, basic aspects of this approach—namely the use of surrogate losses, access to finite data, and interpolation methods—can create conditions that facilitate mutualism , in which all groups benefit from coexistence. For each aspect, we demonstrate through a carefully crafted example how it can act as a mechanism that enables stable coexistence. Here we focus our discussion on surrogate loss as a representative construction; for finite data and memorization, we illustrate the underlying distribution and induced game in Figure 3 , and defer the formal analysis to Section D.6 . Coexistence by surrogate loss. A common approach to empirical risk minimization is to replace the 0-1 loss with a convex surrogate loss, such as the hinge loss used in the classic Soft-SVM algorithm [ 13 ] . This enables optimization, but introduces bias: whereas the 0-1 loss applies uniform penalties, surrogates penalize misclassified points relative to their distance from the decision boundary of h . Our next result uses this artifact to construct a game having two groups that are complementary in their incurred bias, and hence balance each other to induce mutualistic coexistence. Theorem 2 . There exists an evolutionary prediction game in which learning with the hinge loss and \ell_{2} regularization induces a mixed equilibrium that is both stable and fitness-maximizing. The construction is outlined in Figure 3 (Left), and proof is provided in Section D.6 . The proof leverages the bias of hinge loss against minority classes: For each group alone, minimizing the hinge loss biases the learned classifier against the minority class, which is suboptimal. But since each group has a different minority class, these biases cancel out at the mixed equilibrium. The surprising property is that this state is also evolutionarily stable: when one group grows slightly (i.e., p_{1}=0.5+\epsilon ), it loses more from the fact that the other group shrinks (i.e., p_{2}=0.5-\epsilon ) than it gains from its own growth. In this sense, each group needs the other to maintain its existence, giving rise to stable and mutualistic symbiosis. Section D.6.3 expands on this idea to show that varying the amount of regularization leads to bifurcations , in which the game transitions between 3, 1, and 5 equilibria. Interpretation. In the example game above, as in those we construct for finite data ( Section D.8 ; Figure 3 (Right)) and interpolation ( Section D.7 ; Figure 3 (Center)), mutualism is enabled by how user self-selection compensates for the algorithm’s imperfections. When the resulting bias is complementary across groups, this can give rise to (implicit) cooperation. One interpretation is that the existence of each group acts as regularization on others: from the perspective of group k , the objective can be written as \argmax_{h}\mathrm{acc}_{k}(h)+\lambda R(h) where R(h)=\sum_{j\neq k}\mathrm{acc}_{k}^{\prime}(h) is a data-dependent regularizer with coefficient \lambda=\frac{1}{p_{k}}\sum_{j\neq k}p_{j} . Nonetheless, our results should not be taken to imply that proxy losses or finite data are a good means to achieve coexistence; rather, they highlight how mutualism can organically materialize when learning affects different groups differently. We explore this further empirically in Section 6.2 . 5.2 Stabilizing Coexistence Equilibria Thus far we have considered evolutionary prediction games in which learning pursues the conventional objective of maximizing accuracy at each timestep on the current distribution. But if the learner is aware of population dynamics and of how the learned classifier may influence outcomes, then it makes sense to consider learning algorithms that take this into account. The question of course is how. Here we focus on a particular aspect of this general question that arises from our last result in Sec. 4 , namely: given a system with a desirable but unstable mixed equilibrium {\bm{p}}^{*} , how do we stabilize it? We propose a conceptually simple solution that works by reweighing examples in the standard accuracy objective in a way that accounts for dynamics. The main idea is to invert the natural tendency of dynamics to push away from the unstable equilibrium, and instead pull towards it, by training ‘as if’ the distribution was at a different state {\bm{p}}^{\prime} than its actual current state {\bm{p}} : Proposition 2 . Let \mathcal{A}^{\mathrm{opt}}({\bm{p}}) be an oracle algorithm with equilibrium {\bm{p}}^{*} . If {\bm{p}}^{*} has full support (and thus is unstable), then it becomes stable under \mathcal{A}^{\prime}({\bm{p}})=\mathcal{A}^{\mathrm{opt}}(2{\bm{p}}^{*}-{\bm{p}}) . Proof in Section D.9 . The idea is that \mathcal{A}^{\prime} trains ‘as if’ the state is 2{\bm{p}}^{*}-{\bm{p}} , which can be achieved by weighing examples from each group k by w_{k}=\frac{p_{k}}{2p^{*}_{k}-p_{k}} . This inverts the natural tendency of dynamics to push away from the unstable equilibrium, and instead pull towards it. The statement regards optimal classifiers and requires knowledge of {\bm{p}}^{*} , but in practice we can set w according to an estimated \tilde{{\bm{p}}} . We further explore this approach in Section 6.1 . 6 Experiments To demonstrate how some of the principles we have discussed so far apply in more realistic settings, we now turn to explore evolutionary prediction games empirically using real data and simulated dynamics. We include three experiments. The first experiment complements our results on Sec. 5.2 by demonstrating how stabilization can improve both overall and per-group accuracies via coexistence. The second complements Sec. 5.1 , showing how mutualism can arise organically due to group balancing, here from complementarities in label noise. The third experiment explores fairness under population dynamics (Prop. 1 ) to illustrate the limitations of static fairness measures. 6.1 Mutualistic Coexistence From Data Augmentation Data augmentation is a method for enriching the training set with class-preserving transformations of inputs (e.g., camera angle, source of lighting) [ 55 , e.g.,] . Augmentations are typically considered as artificial constructs, but there are also cases where they emerge naturally (e.g., medical images from different hospitals or imaging devices). Here we show that when different sources correspond to different groups, data pooling can serve to foster beneficial coexistence in which all groups gain from the existence of others. This suggests that ‘natural’ augmentations can be effective also in the long run. Method. We use the CIFAR-10 image recognition dataset [ 36 ] , which consists of 60,000 32x32 color images in 10 classes, with 6,000 images per class. As horizontal image flips are considered class-preserving for this dataset, group A consists of images sampled directly from CIFAR-10, and group B consists of images under horizontal flip. We use a ResNet-9 network for prediction [ 26 ] , and train it using the ffcv framework with default optimization parameters [ 38 ] . For each {\bm{p}} , we measure the prediction model’s accuracy on the original images from the CIFAR test set (representing \mathrm{acc}_{A}(h_{\bm{p}}) ), and on their flipped counterparts (representing \mathrm{acc}_{B}(h_{\bm{p}}) ). For stabilization, we use the method described in Section 5.2 with {\bm{p}}^{*}=(0.5,0.5) . We simulate the evolutionary process using discrete replicator dynamics, and use linear interpolation to determine intermediate fitness values. Results. Figure 4 (Left) presents the estimated evolutionary prediction game, averaged across 20{} repetitions. The game has three equilibria: Two stable equilibria with single-group dominance ( 92.6\pm 0.1\%{} accuracy), and an unstable coexistence equilibrium ( 93.5\pm 0.1\%{} accuracy). Although fitness is typically higher for the larger group, the mixed equilibrium attains the highest overall population accuracy, suggesting that each groups benefits from the existence of the other. Reaching this point however requires stabilization; Figure 4 (Center) shows how our algorithm from Section 5.2 is able to attain stable beneficial coexistence, despite using an empirical proxy objective. Appendix F includes additional results on sampling noise, time to convergence, and sensitivity analysis for stabilization. Figure 4: Empirical evaluation in two-group settings. (Left) Game induced by CIFAR-10 with Resnet-9 and groups representing horizontal flips. The game has unstable mutualism ( Section 6.1 ). (Center) Simulated stabilization of replicator dynamics on the CIFAR-10 game ( Section 5.2 ). (Right) Game induced by MNIST with label noise, showing stable mutualism ( Section 6.2 ). 6.2 Stable Coexistence From Overparameterization Modern neural networks are frequently overparameterized, and are known to achieve strong test-time performance despite memorizing the training data [ 64 , e.g.,] . Recent research attributes this phenomenon to the implicit biases introduced by the training process [ 60 , e.g.,] . Building on these ideas, here we demonstrate that stable coexistence can emerge when an overparameterized neural network is trained on data in which groups have complementing label noise. Our construction considers a multi-class settings where each group includes a majority of noisy examples from some class, and a minority of clean examples from other classes. This aims to captures settings in which user groups (e.g., by country) differ in the marginal distribution p(y) over classes (e.g., spoken language), and where the dominant class is more likely to be misclassified (e.g., due to many accents). Method. We use the MNIST dataset, set K=2 , and split the classes unevenly between two groups: Group A is biased towards even digits {\left\{{0,2,4,6,8}\right\}} , whereas group B is biased towards odd digits, both with a 4:1 imbalance. For each group, we introduce label noise to the majority classes, mapping the true label of each digit d to the next digit with same parity with probability 0.2 (i.e. for group A , label noising stochastically maps 0\mapsto 2 , 2\mapsto 4 , etc.). We train a convolutional neural network for 200 epochs using stochastic gradient decent with momentum. The training and test sets are split independently, and for each {\bm{p}} we measure the prediction model’s accuracy on both splits of the test sets (representing \mathrm{acc}_{A}(h_{\bm{p}}) and \mathrm{acc}_{B}(h_{\bm{p}}) ). Under the given parameters, the label noising process leads to an accuracy upper bound of 84\% assuming perfect recognition. Results. Figure 4 (Right) shows the estimated evolutionary prediction game, averaged across 50{} repetitions. Note how label noise has ’flipped’ the game in that accuracy is higher for the minority group (i.e., \mathrm{acc}_{B}>\mathrm{acc}_{A} when p_{B}