A test of time: Modeling the long-term success of crowd collaborations Authors: Abraham Israeli, David Jurgens, Daniel M Romero Published: 2026 ABSTRACT AbstractThe Internet has significantly expanded the potential for global collaboration, allowing millions of users to contribute to projects like Wikipedia. While some of these collaborative efforts see early success, their long-term success is key for lasting impact. Despite its importance, this dimension of success remains largely unexplored. Prior work has assessed the success of online collaborations, however, most approaches are time-agnostic, evaluating success without considering its longevity. Research on the factors that ensure the long-term preservation of high-quality standards in online collaboration is scarce. We address this gap. We propose a novel metric, “Sustainable Success,” which measures the ability of collaborative efforts to maintain their quality over time. We introduce the SustainPedia dataset, which compiles data from 48.7K Wikipedia articles. All articles in SustainPedia have reached the highest indicators of quality provided by the English Wikipedia, but a portion of them (7.5%) were later demoted from this high-quality status. SustainPedia includes each article’s label and more than 300 explanatory features such as edit history, user experience, and team composition. Using this dataset, we develop machine learning models to predict the sustainable success of Wikipedia articles. Our analysis reveals important insights. For example, we find that articles that take to be recognized as high-quality are more likely to maintain thier status over time (i.e. be sustainable). Additionally, user experience emerged as the most critical predictor of sustainability. Our model provides the opportunity of automatically flagging articles at risk of being unsustainable. It also supports actionable factors that are significantly associated with (un)sustainable Wikipedia articles. The ability to sustain high-quality standards over time is essential for the success of online collaborative efforts. This study introduces the concept of “Sustainable Success,” a metric that captures quality longevity in collaborative projects. Using a dedicated dataset that we introduce, of 48.7K English Wikipedia articles, we reveal critical factors of sustainability. Among those, we show that experienced collaborators are crucial for maintaining a sustainable collaboration and that a prolonged and gradual path toward attaining a recognized high-quality status, rather than quick and sudden recognition, is associated with sustainable success. Introduction Collaboration plays a crucial role in societal progress, shaping the way individuals and groups share knowledge, solve problems, and innovate (1–5). In the digital era, platforms like Wikipedia and GitHub have revolutionized collaboration by allowing millions of users to collectively create content (6–10). However, while collaboration is rooted in human behavior, and online platforms empower millions to do so easily, the reality remains that not all collaborative efforts lead to success. Humans have long sought to assess group success across various dimensions and contexts (11–16). In particular, evaluating and predicting the success of collaborative groups is important for optimizing work efficiency and gaining deeper insights into human dynamics. As a result, there has been growing interest in understanding the factors that determine the success or failure of crowd collaboration. Research in this area is primarily motivated by two key objectives: (i) Better predicting the success of online collaborations (17–19) and (ii) Uncovering the underlying mechanisms and factors that foster success (20–22). Advancing this knowledge is essential for improving existing collaborative efforts to maximize efficiency and impact. However, proposed approaches focus on either the success of communities as a group of people (11, 23, 24) or the quality of collaborative efforts’ output at a single point in time (25). The latter are overlooking the importance of sustained success over the long term, which necessitates continuous effort and ongoing maintenance by the collaborative team to preserve the high quality of their shared output. While the importance of maintenance, repair, and long-term care has recently been recognized in innovation studies (26), it has yet to be fully explored in the context of collaborative efforts. Sustained success in collaborative environments is a challenge. Early signals of success do not always guarantee long-term viability. Prior work explored factors that influence the persistence of collaborative efforts, such as governance structures (24, 27), contributor retention (23, 28), and social dynamics (29, 30). However, none of these studies have explored the capacity of collaborative efforts to maintain a high-quality status over time—i.e. avoid an initially successful collaboration falling apart. To address this gap, we introduce a metric called “Sustainable Success,” which assesses this aspect of success directly—the ability of collaborative teams to maintain high-quality standards over time. Using the new sustainable success measure, this study asks the following two research questions: (RQ1) What accuracy can we achieve in predicting the sustainable success of an online collaboration?; and (RQ2) What factors contribute to the sustainable success of an online collaboration? To answer these questions, we explore collaborations on over a decade of behavior in the English Wikipedia. Through large-scale human collaboration, Wikipedia became one of the biggest open and community-driven knowledge repositories available online (31). Wikipedia has a rich collaborative environment, with various means of user interaction, including voting, discussions, and communication. Crucially, Wikipedia editors also directly evaluate its content in a structured way (32–35), providing a reliable basis for measuring sustainable success. The public accessibility of Wikipedia’s data allows us to comprehensively explore the factors that contribute to the sustainability of collaborative efforts, thereby answering our RQs in a thorough and reliable way. Modeling unsustainability In our study, we explore Wikipedia’s two most trustworthy content assessment levels: Featured Article (FA) and Good Article (GA). The two tags signify high-quality articles that meet specific editorial standards (36, 37) and have been widely used in Wikipedia-related research (19, 34). FA represents the pinnacle of quality, having undergone rigorous peer review and meeting the strictest criteria, such as comprehensive coverage, neutrality, and well-researched content. GAs, while not as rigorous as FAs, also meet substantial quality standards, including being well-written, factually accurate, and verifiable. As of September 2023, just 0.1 and 0.6% of English Wikipedia articles are tagged as FA and GA, respectively. FAs are marked with a bronze star symbol while GAs with a green plus. Both tags highlight Wikipedia’s best work, reflecting reliable, in-depth content. The community’s judgment of quality is not permanent, however. The FA and GA statuses can both be removed if the article no longer meets the community’s quality standards, effectively demoting the article. Figure 1 illustrates this promotion and demotion process. Note that promoted articles may be demoted and subsequently repromoted, as in Fig. 1, reflecting the dynamic nature of Wikipedia’s quality assessment process. We leverage this ongoing quality assessment to model the sustainable success of Wikipedia articles. Article lifecycle example. (Top) Major events of the Wikipedia article about the edible fruit Durian. (Bottom) A screenshot from the talk page of the Durian article, including both its promotions and its demotion milestones. The article kept its promoted state for more than 15 years before being demoted in 2022. Source: Wikipedia, “Durian,” Wikimedia Foundation, licensed under CC BY-SA 4.0. https://en.wikipedia.org/wiki/Talk:Durian (accessed 2026 August 11). We use all articles that, by September 2023, have reached one of these two highest quality levels at some point in their lifecycle to build SustainPedia. SustainPedia includes 48.7K Wikipedia articles which we use for our research and make publicly available for further research. Notation We use the following notation in the rest of the article. A is the set of Wikipedia articles we process. For each article a∈A, we mark to the following three timestamps: tbirtha, tproma, tdema. The first refers to the time when the article was created (i.e. its first revision), and the other two refer to the time when the article was promoted and demoted, respectively. Some articles lack tdema as they were never demoted. We mark dproma as the duration an article kept its promoted state, measured in years. For demoted articles, this is tdema - tproma. We define the following indicator for each article a∈A: Note that in a minority of cases (19.4%), articles are repromoted after being demoted. Since our research objective is to find what makes an article sustainable, any demotion makes the article unsustainable, even if it was later repromoted. Therefore, we mark 1unsustaina=1 if article a was demoted at least once in its lifecycle. We frame the problem as a time-to-event (survival) regression task. We utilize these survival models to predict the duration an article a maintains its promoted status before a demotion event occurs (if at all). The primary variable of interest is the survival time, denoted as T. For each article a∈A, we define Ta as the duration (in years) between an article’s promotion and either its subsequent demotion or the end of the observation period (censoring).a Note that in these settings, Ta≜dproma. Our models estimate the hazard rate associated with an article based on information available at the time of its promotion (tproma). Specifically, the model learns how article characteristics, measured at the promotion time of the article, relate to the future risk of demotion. This approach naturally incorporates the duration an article remains in a promoted state (dproma) without requiring an arbitrary time threshold, thus preserving the continuous nature of the data and accounting for “right-censored” articles that have not yet been demoted. Ta paired with 1unsustaina allows the model to distinguish between confirmed failures and ongoing sustained articles. To understand the process behind sustained success, our models use different categories of features related to creating and maintaining the article. While constructing the different features, we only consider available data before tproma. This modeling design choice is motivated by our goal to identify early signals of collaborative communities that indicate (un)sustainability. Post-promotion data may offer additional insights. However, it is less practical for predictive purposes, as it introduces uncertainty regarding the data collection cutoff point and is beyond the scope of our research. Specifically, we build the following six feature sets (n denotes the number of features per set): network structure of collaborators (n=27), topics that the article is associated with (n=247), team composition (n=5), discussion patterns (n=36), edit history (n=7), and the experience of the article’s collaborators (n=4). The latter, which was found to be the most significant feature set, is calculated based on the previous participation of users in successful collaborations. To summarize, our two main contributions are: We introduce a novel definition and measurement of continued collaborative success. We define it as the ability to keep high standards over a long enough time, which we name as sustainable success. We introduce a novel definition and measurement of continued collaborative success. We define it as the ability to keep high standards over a long enough time, which we name as sustainable success. We highlight the most important factors that are associated with sustainability, such as team members’ experience. These factors are extracted from a machine learning model that we introduce, which predicts the sustainable success of Wikipedia articles. Importantly, our findings suggest that the factors previously found to be associated with initial and general success differ from those identified in our research. We highlight the most important factors that are associated with sustainability, such as team members’ experience. These factors are extracted from a machine learning model that we introduce, which predicts the sustainable success of Wikipedia articles. Importantly, our findings suggest that the factors previously found to be associated with initial and general success differ from those identified in our research. In addition, we introduce a new dataset named SustainPedia, containing all the information we used for analysis and modeling. We make SustainPedia and the code we developed public to be further used by the research community.b Next, we introduce the results and findings of our research. For further details about the methods and data used, please refer to Methods and Data sections that we present toward the end, after the Discussion section. Results We are interested in predicting the risk level of a promoted article a∈A becoming unsustainable. That is, we quantify the hazard rate of a being demoted from its high-level quality assessment in the future. To do this, we employ a survival analysis framework trained to predict a continuous risk score, allowing us to model the underlying “pressure” toward demotion as a function of time and article characteristics. We further elaborate on the methods and experimental settings in the Methods section. We experiment with an array of algorithms ranging from simple linear survival regression models to more sophisticated survival ensemble models. We do not expect complex deep learning and transformer architectures to outperform standard ML models in our settings, since we formulate SustainPedia as tabular data (38).c We report only the results obtained from the Gradient-Boosted-Decision-Trees (GBT) model (adjusted for censored data), which outperforms linear models. We use the scikit-survival Python package (39) implementation of the algorithm, with the default parameters of the package. Specifically, we build the model with 100 estimators (i.e. trees). We execute all algorithms in an ablation manner to learn the importance of the different feature sets. We make the code developed as part of the research and the SustainPedia data public in our project’s GitHub repository. Survival model performance We visualize the model performance of each of the six features set in Fig. 2. We also include a “Baseline Model” performance, which is based on a single explanatory feature model—“Num-of-Revisions-Normalized”. Further details about this feature can be found in the Data section, under the “Edit History” features explanation. We evaluate model performance using the Concordance Index (C-index) (40), which measures the model’s ability to correctly rank the predicted survival times of article pairs. A C-index of 0.5 indicates performance no better than random chance, while a value of 1.0 represents perfect agreement between predicted risk and the actual chronological order of demotion events. Model’s performance. We present the aggregate values from the 5-fold cross-validation evaluation process. Boxplots represent the confidence intervals around the average values. The C-Index y-axis is the Concordance-Index evaluation measure. A random model performance is represented by the dashed horizontal line. We also include more detailed results of the models’ performance in Table S1 under the Supplementary material, with an additional evaluation measure we record per experiment. The best performing model, which uses the entire feature set, achieves a C-Index of 0.79 and 0.74 over the FA and GA, respectively. In comparison to both a random model and a baseline model that relies on a single feature, it demonstrates substantial improvements in performance. Using each feature set independently yields statistically significant improvement compared to a random model.d In both FA and GA use cases, combining all feature sets (i.e. “All”) leads to statistically significant improvements, outperforming any individual feature set. The “User Experience” feature set is statistically better than other single feature sets, for the FA use case. The same holds for the GA use case, except that it is not significantly better than the one other feature set (i.e. “Edit History”). We note that while trailing behind, the “Edit History” feature set alone achieved a significant improvement over the baseline model in both use cases. On the other hand, the “Topics” feature set, although comprised of a large set of features (n=247), significantly underperformed compared to other models. Next, we analyze the data and the best-performing model. We present results for FAs. However, we observe similar patterns for GAs. Prediction analysis We further performed a post-hoc analysis of the model’s predictions, specifically examining articles identified as highly vulnerable by the best-performing model. We define vulnerable articles as those characterized by a high predicted hazard rate despite not having experienced a demotion event during the observation period. This is analogous to a false positive analysis in traditional classification. To formalize this, we define a threshold δ for the predicted hazard; specifically, we analyze two distinct subsets: (i) the top 100 articles with the highest predicted hazard scores and (ii) articles with a positive hazard prediction (δ>0). This dual-threshold approach allows us to examine both the most extreme cases of predicted unsustainability and the broader set of articles that the model identifies as being at a relative elevated risk. Vulnerable articles are interesting as they can shed light on articles that have not officially been demoted but are possibly at risk. To further investigate vulnerable articles, we look at articles with the highest predicted hazard rate according to our model, but not currently demoted. We use the Wikipedia reviewing system to analyze this set of articles. As illustrated in the bottom table in Fig. 1, an article can go over multiple review types throughout its lifecycle. The example article (Fig. 1), “Durian,” had three distinct review types. Note, that we use these reviews to identify promoted and demoted articles, but we do not use data from the review system to construct our prediction features. Not all FA (or GA) reviews lead to a demotion. In many cases, the article keeps its existing status. However, even the fact that an article goes over a review is a signal that the community identified potential issues with the article.e We use the review information to see if FP articles tend to be reviewed more frequently. Figure 3 highlights that our hypothesis truly holds. Vulnerable articles (second right-most bar) are being suspected by the Wikipedia community—reflected by a statistically significant higher number of reviews compared to the sustainable articles population. Note that unsustainable articles have at least one review since they were demoted at least once. We reiterate that the prediction model is not exposed to the review information at all. The model is fed only with information in the [tbirtha,tproma] timespan. The reviews we analyze here occurred after tproma. Average number of reviews per population, FA use case. The vulnerable articles are reviewed significantly more frequently compared to the sustainable articles population, which highlights their demotion potential—identified by the Wikipedia community. Naturally, the number of reviews for the unsustainable population is the highest since these demoted articles go over at least a single demotion. As anticipated and reflected in Fig. 3 (second left-most bar), using the top-100 vulnerable articles reflects an even stronger trend, with the average number of reviews for the top-100 vulnerable articles reaching an average of 0.8 reviews per article. Additionally, Table S2 provides a list of the top ten articles from the vulnerable articles population, and Table S3 lists the top 10 articles with the highest predicted hazard rate that were recently promoted. According to our model, these articles have a high probability of being unsustainable. We expect that these articles are at high risk of demotion. Model interpretation To support RQ2, we use the SHAP framework (41) to shed light on the specific features that impact the model’s prediction. A SHAP value is calculated for each explanatory variable (feature) and input instance (i.e. an article in our case) for a specific model. A high (low) SHAP value indicates the positive (negative) impact of the feature on the prediction of the specific instance. Looking at the aggregate values for a specific feature provides a way to understand the importance of features and their positive/negative impact. The absolute SHAP values of the 10 most prominent features are presented in Fig. 4. The bar color and pattern correspond to the feature’s positive or negative impact on the prediction. Note that we predict hazard rates, and hence a positive (negative) SHAP value indicates a high (low) impact on the risk level of being unsustainable. Top 10 SHAP feature importance rankings. The majority of feature sets are represented in the top 10 list of features, with the Experience set dominates the list. Orange (top-right to bottom-left lines) bars indicate a negative correlation with the hazard rate; specifically, these orange bars represent features that decrease the risk of demotion, thereby contributing to article sustainability, while blue bars indicate increased vulnerability. Experience is the teacher of all things Aligned with the results in the first part of the results, the Experience features set plays the most crucial part in the model. Three out of the 10 most prominent features are from the “Experience” features set. The negative impact (i.e. orange colored bars) reflects that low experience values (i.e. inexperienced team members) are associated with unsustainability. Excellence takes time Three out of the ten dominant features are related to the time it takes an article to achieve a promotion. Those are Time-to-Promotion, Was-a-Good-Article, and Num-of-Revisions-Normalized. The first two out of those (and the more dominant ones) indicate that unsustainability is negatively associated with a long progress that takes time. Time-to-Promotion is simply tproma−tbirtha (measured in days) which indicates that fast-promoted articles, in terms of time, are more in a risky situation to fail in the future (i.e. Data section) points out that articles which did not go through a two-step promotionf are more in a risk to be unsustainable, showing again that a slower trajectory to FA with an initial stop at the GA status is more likely to lead to a sustainable article, in FA. 19.4% of the articles that were directly promoted to FA are found to be unsustainable, compared to only 4.8% of those that were first promoted to GA. Marginal but critical The rest of the features come from two other feature sets—“Team Composition” and “Discussions.” Collaborators-Gini negative impact on the model highlights that a lower Gini (equal contribution) among collaborators is strongly associated with unsustainability. This is aligned with our findings based on the Association Heatmaps (see further explanation in Section S1.D as well as in the left-most heatmap in Fig. S4, both under the Supplementary material). The Median-Formality-Per-User, the Median-Toxicity-Per-User, and the Positive-Sentiment-Percentage shed light on the importance of discourse patterns among collaborators. These features are extracted from each article’s Talk Page and quantify the overall emotional tone and the concentration of harmful language expressed by individual contributors. Low formality (third-to-last bar) and persistently negative sentiment (final bar) are associated with an increased hazard rate, significantly shortening the expected duration of an article’s promoted duration. In this survival framework, these features serve as indicators of collaborator conflict or dysfunction, which accelerate the transition toward demotion. This suggests that articles characterized by less formal or more critical discourse are at a greater risk of failing to maintain their high quality status over time. Conversely, Median-Toxicity-Per-User exhibits a counter-intuitive relationship with longevity; the model identifies it as a protective factor associated with a reduced hazard rate. This suggests that while toxic discourse is generally detrimental, in high-stakes quality maintenance, it may act as a proxy for rigorous gatekeeping or intense communal engagement that paradoxically prevents quality decay. Additional validation of the model performance is provided in the Supplementary material. Specifically, we include: (i) An evaluation of the model robustness over long-term promotion horizons (Section S1.C); (ii) An analysis of the features used by the model (Section S1.D), including heatmap visualizations that explore meaningful interaction effects between explanatory features; and (iii) A risk-stratification analysis examining the relationship between model-predicted risk and observed demotion outcomes (Section S1.E and Fig. S5). This analysis reveals a monotonic increase in observed demotion rates across the predicted risk spectrum, providing additional evidence that the model effectively distinguishes between articles at low and high risk of future demotion. Discussion The objective of most related studies is “Success Prediction.” That is, to research what leads crowd collaboration to success in general. Our objective is different. We aim to measure these collaborations through the lens of whether success holds over time, using our proposed “Sustainable Success” measure. We see these two objectives as different. Naturally, all collaborative efforts we analyze were successful at some point in time. However, some of them were able to maintain their success while others were not. “Sustainable Success” is focused on this distinction. Highlighting the similar (and different) dominant features between “Success Prediction” and “Sustainable Success” Prediction is insightful. We do find interesting differences between the factors that lead to each of these two. A key feature that we find to be associated with “Sustainable Success” is a prolonged and gradual path toward attaining a recognized high-quality status. This feature is unique to our study and has yet to be recognized in other “Success Prediction” studies. Naturally, “Success Prediction” studies are limited in the way this feature can be constructed as their target feature is different than ours. The participation of experienced team members with a proven history of involvement in successful collaborations, which we find statistically significant feature, was also found to be statistically significant in a few “Success Prediction” studies (42, 43). For other features, such as number-of-collaborators and most network features, while we did not find those to be the most dominant, other “Success Prediction” studies found them as significant predictors (44–47). Overall, our findings emphasize the unique features critical to sustainable success, while also identifying shared elements with existing research on the general task of success prediction. Experience features dominance Across all our experiments and analyses we had, the different experience-related features were found to be most impactful. This highlights the importance of intellectual capital while building a collaborative team (48, 49). Specifically, having Wikipedians who are less experienced in editing and contributing to Wikipedia projects puts articles at risk of becoming unsustainable. We believe that experienced Wikipedians play a central role not only in driving articles to excellence and maintaining that quality over time but also in selectively choosing to which collaborations they are aiming to join. This may reflect both their editorial skills and their strategic decision-making. One possible explanation for the predictive strength of the experience-related features is that seasoned editors are more familiar with Wikipedia’s promotion/demotion process as well as quality standards, allowing them to more effectively navigate (or even influence) these processes. In addition, experienced editors may possess institutional memory of prior conflict resolutions and editorial challenges, equipping them with the knowledge and tactics necessary to keep articles in high quality standards (e.g. FA and GA) and avoid their demotion. Another important element is that the English Wikipedia also has a “Manual of Style”g for FAs and GAs. This allows those who know it well and how to translate it into writing articles (i.e. most likely experienced Wikipedians) an advantage. According to our findings, future research could further focus on the role of experienced Wikipedians in shaping (un)sustainable articles. A deeper understanding of why experienced editors choose to join specific collaborative efforts—and whether they consider the sustainability potential of those efforts, may provide valuable insights into the mechanisms that are behind the experience-related features. The broader picture We use Wikipedia as a testbed to answer our research questions. However, quantifying the sustainable success of crowd collaborations goes beyond Wikipedia articles and extends to other online and offline collaborative efforts such as crowdsourced open-source software and entrepreneurial teams (50–53). These crowd-collaboration projects span multiple domains and play a critical role in large-scale projects. In such collaborations, success is not merely determined by the short-term success or the initial hype they make but rather by the longevity and lasting impact they generate. The social dynamics driving these collaborations share parallels with the challenges faced in maintaining the long-term quality and relevance of Wikipedia articles. Actionable features While not all the dominant features we identified are actionable, many are. Notably, a gradual progression toward attaining recognized high-quality status was found to be strongly associated with sustainable success. This finding suggests that implementing a policy of incremental quality assessments, rather than immediate promotion to the highest assessment level, may be beneficial. In the context of Wikipedia, this approach is exemplified by initially designating articles as GA before considering them for an FA status. This can also be applied to lower assessment levels such as A, B, or C, where currently no gradual progression restrictions are applied. Additionally, our analysis highlights the critical role of experienced collaborators with a demonstrated track record of success. This actionable feature should motivate collaborative teams to engage with experienced users and convince them to join their collaborative effort to enhance the likelihood of achieving sustainable success. Although Wikipedia administrators have invested considerable effort in supporting newcomers through the involvement of senior editors (54, 55), less attention has been directed toward incentivizing experienced Wikipedians to contribute to high-quality articles. Our research indicates that such engagement could positively influence the long-term sustainability of articles. Beyond the actionable items we mentioned, our model offers a practical auditing tool for Wikipedia maintainers. By prioritizing the “vulnerable” articles identified in our analysis (i.e. the cases with the highest predicted hazard), Wikipedia administrators can proactively review and stabilize high-risk content before a formal demotion event occurs. This shift from reactive demotion to preemptive maintenance demonstrates the direct application of hazard modeling. Limitations This study focuses exclusively on English Wikipedia articles, limiting its scope to this type of Wikipedia collaboration. By nature, the collaboration teams we analyze are large ones. Additionally, the data used is restricted to the point in time when the study was conducted, meaning that articles may be demoted in the future, which we cannot account for. Our analysis is based solely on Wikipedia’s internal dynamics, without incorporating interactions that may occur outside of the platform (e.g. social media). In addition, given the nature of demotions on the English Wikipedia, SustainPedia contains a small number articles that were actually demoted during the timeframe when data were collected. We further analyze how this impacts the classifier’s performance in the “Supplementary material” (see Results Supplementary Material, part D). We anticipate improved performance with the expansion of the dataset, particularly through an increase in the number of unsustainable examples. Another limitation of our study is that we primarily evaluate feature sets independently, without explicitly modeling potential interaction effects—particularly those involving the “Topic” feature set. While these features may appear less predictive on their own, they could interact with others and still mask interaction effects that vary by domain (i.e. WikiProject in the Wikipedia system). Future work could benefit from modeling such interactions, especially given their relevance to specific collaborative effort domains. In addition, our work does not consider the effect of coordinated editor campaigns to promote or demote articles, or more generally, patterns in how promotions and demotions are distributed in time. Future work could further explore how mechanisms such as coordinated campaigns or waves of promotion or demotion impact the (un)sustainability of Wikipedia articles. Finally, our research is based on naturally occurring data rather than a designed experiment. Hence, the associations identified in this study should not be interpreted as causal. Methods We investigate the sustainability of successful online collaborations by focusing on the sustainable success of English Wikipedia articles. English Wikipedia maintains a well-structured and documented quality assessment system in which each article is assessed with its quality level(s). Wikipedia’s quality assessment system Articles are assigned quality scores in Wikipedia in two ways. First, Wikipedia maintains two quality assessment classes where an article can be tagged as FA or GA. FA and GA determinations are made using Wikipedia promotion cycles that involve whoever is interested in participating, while decisions are made using a Wikipedia consensus. FAs are “the best articles Wikipedia has to offer” (36) while GAs meet “a core set of editorial standards” (37) but are not featured article quality. Second, quality grades (e.g. B, C, Start) are assigned by WikiProjects, which are groups of thematically related articles, such as the “Plants” Wikiproject. Most articles are associated with at least one WikiProject who each assesses an article’s quality separately using the same grading scale. However, each WikiProject can have different quality standards, grading systems, responsiveness to changes in an article, etc. It is common for an article to have different quality grades across WikiProjects (56). In contrast, FA and GA quality assessments are judged Wikipedia-wide using a systematic peer-review evaluation, which excludes the editors of the evaluated article, making them more reliable than WikiProjects scores. Thus, in this study, we only use FA and GA quality assessment levels. As of September 2023, there are 6,345 and 43,556 FAs (36) and GAs (37), respectively. Promotions and demotions Interestingly, both FA and GA quality assessment levels are dynamic. It is common for an article to first be promoted to GA and then later to FA. Additionally, promoted articles can be demoted (and later be repromoted, and so on). As illustrated in Figure 1, the example article was first promoted to GA, then to FA before being demoted in June 2022. Demoted articles are reassigned with an assessment level of Start, C, B, or A (i.e. FAs are demoted to a lower level than GA). The main reasons to demote an article are incomplete/updated content, as new information becomes available, or the article no longer meeting Wikipedia’s criteria for FA or GA. The demotion of an article, the same as promotion, is determined through a consensus derived through discussion at a dedicated Wikipedia discussion page. The reevaluation of an article’s quality is done through an “article review” process, which usually takes a few weeks (57, 58). During this time, the members of the relevant community are allowed to comment on the required improvements and directly edit the article to improve it. Figure S3 illustrates how a review process is announced. A bold comment appears on the “Talk Page” of the relevant article, asking the Wikipedia community to comment about the issue and motivating Wikipedians to edit the article and improve its content. In this research, we use the promotion/demotion processes in Wikipedia to learn what makes successful collaborations sustainable. We define sustainability as maintaining high-quality standards over time. In Wikipedia, this means not being demoted after an initial promotion. For the reasons mentioned above, we only rely on the FA and GA quality assessment levels. Prediction model We employ survival analysis to model the sustainability of promoted articles, a∈A, by estimating the duration of time elapsed before a demotion event occurs. This framework allows us to analyze the hazard rate—the instantaneous risk of an article losing its promoted status, as a continuous function of its characteristics and the time spent in a high-quality state. For each article, we define the primary outcome as the time-to-demotion, Ta, measured in years. We develop independent survival models for FA and GA to account for the distinct quality standards and maintenance dynamics inherent to each. A key advantage of this approach is its ability to handle right-censored data; that is, articles that remain in a promoted state at the end of the observation period are not excluded, but rather contribute to the model by indicating sustainability up to their last recorded timestamp. Our modeling approach possesses the following properties: Demotion: We tag an article as unsustainable if it was demoted at least once in its lifecycle. In rare cases, articles are repromoted after being demoted. However, our research objective is to find what makes an article sustainable over time, and hence, any demotion makes the article unsustainable, even if it was later repromoted. Nevertheless, repromotions are not very common. Out of the demoted articles, only 18% were repromoted. Demotion: We tag an article as unsustainable if it was demoted at least once in its lifecycle. In rare cases, articles are repromoted after being demoted. However, our research objective is to find what makes an article sustainable over time, and hence, any demotion makes the article unsustainable, even if it was later repromoted. Nevertheless, repromotions are not very common. Out of the demoted articles, only 18% were repromoted. Handling of right-censored data: A defining characteristic of our dataset is that articles are promoted at different points in time and do not share a common promotion date. Consequently, our observation of an article’s sustainability is right-censored: for many articles, the “demotion event” has not yet occurred by the end of our study period. Unlike traditional modeling approaches that might require excluding recently promoted articles due to “insufficient observation time,” our survival analysis framework explicitly incorporates these cases. Each article a contributes to the model through its observed duration Ta, regardless of whether that duration ends in a demotion event or is simply cut off by the current date. This allows us to utilize the entire corpus of promoted articles without imposing an arbitrary survival threshold, thereby maximizing the statistical power and representative nature of our analysis. Handling of right-censored data: A defining characteristic of our dataset is that articles are promoted at different points in time and do not share a common promotion date. Consequently, our observation of an article’s sustainability is right-censored: for many articles, the “demotion event” has not yet occurred by the end of our study period. Unlike traditional modeling approaches that might require excluding recently promoted articles due to “insufficient observation time,” our survival analysis framework explicitly incorporates these cases. Each article a contributes to the model through its observed duration Ta, regardless of whether that duration ends in a demotion event or is simply cut off by the current date. This allows us to utilize the entire corpus of promoted articles without imposing an arbitrary survival threshold, thereby maximizing the statistical power and representative nature of our analysis. From GA to FA: Many FAs are first promoted to GA status (see Figure 1), and these articles are utilized to model both FA and GA survival processes. For example, the article illustrated in Fig. 1 is used in the hazard estimations for both the GA and FA trajectories, as the demotion event marks the end of its survival duration in both states. From GA to FA: Many FAs are first promoted to GA status (see Figure 1), and these articles are utilized to model both FA and GA survival processes. For example, the article illustrated in Fig. 1 is used in the hazard estimations for both the GA and FA trajectories, as the demotion event marks the end of its survival duration in both states. Model evaluation We evaluate the survival regression models using 5-fold cross-validation. To analyze article-level trends, we use the article-level predictionsh for error analysis and model interpretability. We report the Concordance Index (C-index) (40) as well as the Cumulative-Dynamic-AUC (59), which extends the AUC to a survival context by evaluating the model’s discriminative power at specific time points. This metric specifically measures how well the model distinguishes between articles demoted by time t and those that remain sustainable. Data Each Wikipedia article maintains its content page (the actual article) and its talk page. In this research, we make use of both pages. While the content page and its meta-data contain essential information on how the article’s content is built and by whom, the latter contains discussions between the community members. We collect the data of all relevant articles using the Wikipedia dump files, which are monthly released by the WikiMedia Foundation (60). We adhere to ethical standards, ensuring that Wikipedian identities are not disclosed or analyzed in our models. We only take into account aggregated measures of users’ activity. The Wikipedia dump files allow us to collect historical information about each article. This way, we can track and collect information about each revision of both the content page and the talk page. The dump files contain detailed information per revision, including time, editor name, textual content, etc. We use the September 2023 release of the English Wikipedia. Most talk pages contain helpful information about the article. Among those are the “Article Milestones” that include FA or GA promotions and demotions, as well as the WikiProjects associated with the article. We collect data for both FA and GA use cases in the same way to build the SustainPedia dataset. SustainPedia contains all the information we used for analysis and modeling. We make SustainPedia public to be further used by the research community for future research on sustainable crowd collaboration. In the rest of this section, we explain how we construct the SustainPedia population, the target feature, and the explanatory features. SustainPedia population To construct the SustainPedia population, we look for articles that are in the FA or GA status as of September 2023, or have been in that status before September 2023, including those that have been demoted. Wikipedia maintains lists of existing FAs and GAs (36, 37). In addition, there are Wikipedia pages that list demoted FAs, named “Former Featured Articles,” and demoted GAs, named “Delisted Good Articles” (61, 62). We use these four lists to construct the set of articles in SustainPedia. Target feature construction To construct the target feature per a∈A we extract its promotion to FA and/or GA information. We also extract its demotion information, if any. To do so, we use two complementary methods: article milestones and Wikipedia templates. Many Wikipedia articles maintain a table on their talk page with information about the article’s milestones. Usually, this table contains the dates of promotion, demotion, and reviews that the article has gone through. Figure 1 (bottom) contains a screenshot of the article milestone section of “Durian”.i We use this information to extract both the promotion/demotion type (FA or GA) as well as the date of promotion/demotion. Relying solely on this information source is deficient. Having an “article milestones” section is not mandatory, and in some cases, the section exists but lacks some of the required information. Therefore, we use a second method to extract promotions/demotions. In Wikipedia’s HTML content, FAs and GAs are tagged using specific templates. The FAs and GAs are tagged using the {{Featured Article}} and {{Good Article}} templates respectively. Once an article is promoted (demoted), the relevant template is added to (removed from) the HTML of the article. Since we process all revisions of each a∈A, we can extract the revisions (and dates) in which an article was using the GA/FA template to identify if the article as promoted or denoted accordingly. However, any Wikipedian can add and remove these templates. Edits that do not respect the assessment evaluation process, e.g. due to vandalism or lack of familiarity with Wikipedia guidelines, should be considered when applying this methodology. Hence, we track the changes in the FA and GA templates over time and consider a promotion/demotion only if at least one of the following criteria is met: The change holds for at least five sequential revisions. The change holds for at least five sequential revisions. The change holds for at least a month (i.e. 30 days). The change holds for at least a month (i.e. 30 days). The thresholds we applied above (e.g. 30 days) are intentionally stringent to ensure that only reliable changes are considered. Overall, the vast majority of promotion and demotion events (98.2%) are identified using the article milestone approach, with only a small fraction detected through the template-based method. Table S4 presents basic data statistics for SustainPedia. In addition to the significant variance observed among articles across various characteristics (e.g. Revisions), we find that the classification problem is heavily imbalanced. Specifically, only 16% (6%) of the FAs (GAs) have ever been demoted. While modeling the problem, we consider articles that were not able to keep their promoted state for long enough as unsustainable. This leads to an even more extreme imbalanced problem. Explanatory features We consider a large set of explanatory features for the prediction task. While constructing the different features, we only consider available data before tproma∀a∈A. We derive six distinct feature sets, characterized in detail in the rest of this section. A comprehensive summary of these feature sets, including variable types, is provided in Tables S5–S8. Edit history features (n=7) This feature set, named EH, captures the editing dynamics of the article, which was studied from different perspectives (23, 63, 64) and have been shown to be associated with performance in Wikipedia (65, 66). We extract and calculate EH from the article’s revision history in the [tbirtha, tproma] timespan. It includes, number-of-collaborators (EH1), number-of-revisions (EH2), time-to-promotion (EH3), and reverted-revisions-percentage (EH4) which is the number of reverts out of EH2. We detect reverts by comparing the SHA1 value of each revision against all previous text revisions.j The SHA1 is recorded per Wikipedia article revision as part of its metadata information. We also create normalized features for the first two metrics to account for edit activity and time: EH1/EH2 and EH2/EH3, respectively. The Was-a-good-article (EH5) binary feature indicates whether the article was first promoted to GA before a second promotion to FA (see example in Fig. 1). It is only relevant for FAs. Network features (n=27) Numerous scholars show the effect of network structure among collaborators and success (67–69). In Wikipedia, a few studies focus on the network structures and their impact on Wikipedia articles (70–73). We refer to network features as the ones extracted from the edit-network Ga=(Va,Ea) that we construct for each a∈A. Ga is based on the edit interactions among collaborators. Va (nodes) and Ea (edges) represent Wikipedians and their interactions, respectively. Note that anonymous editors are excluded while building the network. Inspired by recent studies that introduce network structure to Wikipedia editors (22, 72), we build Ga according to the sequential revision history. A directed edge e∈Ea is assigned between two Wikipedians (v1,v2)∈Va if v2 revised article a, directly after a revision by v1. We extract the following for each Ga: |Va|, |Ea|, number-of-triangles, density, number-of-connected-components (regular and strongly), is-biconnected, and number-of-nodes-to-cut. The latter two are calculated based on the largest component of Ga, and the last represents the smallest number of nodes that must be removed to cut Ga into two connected components. In addition, we calculate the following metrics for each v∈Va: in-degree, out-degree, centrality, betweenness, and closeness. For each metric, we calculate the average, median, and SD over Va to have aggregated features per article. Topic features (n=247) Existing scholars highlight the relation between Wikipedia topics to their edit frequency and edit wars (74, 75). We hypothesize that incorporating topical features will improve the model’s prediction. Adding these features allow the model to incorporate the topic of each article into account while predicting unsustainability, as well as making more complex relations based on the interactions of topic and other features. As we noted previously, most Wikipedia articles are associated with multiple WikiProjects. Each WikiProject rates the importance of the article to the WikiProject as “Low,” “Mid,” “High,” or “Top” (56). For example, “Durian” is associated with twelve WikiProjects. Among them are the “Plants,” “Malaysia,” and “South Asia” WikiProjects in which Durian is associated with a “High,” “Mid,” and “Low” importance, respectively. We utilize this information to generate topical features for a∈A, using a Likert scale ranging from 0 to 4. A zero value is assigned when an article is not associated with a WikiProject. A value of four is assigned when an article is associated with a WikiProject as “Top-Importance.” There are over 2,000 English WikiProjects, and many of them are very small (76). We only use the top 250 by number of articles. Three of the 250 topics were manually removed as they were not purely topical. Those three are: “Articles for creation,” “Spoken Wikipedia,” “Guild of Copy Editors.” 97.7% of the articles in the SustainPedia dataset belong to at least one of these 247 WikiProjects. In these settings, there are at least 100 articles per topic (i.e. WikiProjects) in both FA and GA use-cases. Discussions features (n=36) The way Wikipedians handle their discussions has been shown to be correlated with various success measures and the decisions taken by a discussing group (33, 77, 78). We expect to find a significant correlation also with sustainable success. For example, we assume that polite and respectful discussions would be positively associated with sustainable success since these types of discussions may be more likely to incorporate diverse perspectives. We collected all historical discussions between Wikipedians (i.e. discussers) on the talk page of each article and constructed two types of discussion features. The first type are linguistic features. We use pretrained transformers models to measure those. Specifically, we measure the sentiment (79), formality (80), politeness (81, 82), toxicity (83), and certainty (84) for each comment in a discussion. We use existing implementations for each of these models, publicly available to download and use on Hugging-Face or GitHub repositories.k We then aggregate each, using the average, median, and percentage of high values. We use previously identified thresholds to define high values using each model. To account for uneven contributions across the discussers of talk pages that might bias our measures, we also calculate the mean values for each discusser and then aggregate these values across all discussers using the mean and median. The other type of features are related to the structure of the discussions. Among those 15 features are number-of-discussers, mean-responded-comments, and the discussers-gini (85), which is a measure of statistical dispersion commonly used to measure inequality. A Gini index of 0.0 indicates perfect equality where all collaborators contribute the same number of revisions in the case of articles or comments in the case of discussions, while 1.0 indicates perfect inequality where a single individual generates all content. Experience features (n=4) We hypothesize that the experience of an article’s collaborators positively impacts its sustainable success. To construct “Experience” features, we extract the set of collaborators,Va, of each article a∈A during [tbirtha, tproma]. Va in the rest of SustainPedia (Only up to tproma), excluding article a. Note that this measures experience in editing other articles that were eventually promoted. We construct aggregated measures per article a by either (i) Summing all edits or (ii) Weighing the number of edits by the proportional contribution of each v∈Va. This way, higher-contributing editors are given greater weight in the aggregated measure. Inspired by Stein and Hess (43), we construct two modified features named “Credibility” that consider whether the other articles edited by authors in Va are sustainable articles or not. In this setup, edits to sustainable (unsustainable) articles positively (negatively) contribute to the aggregated measure. We only count sustainable (unsustainable) cases if the promotion (demotion) is known by tproma. As before, aggregation over Va is done by either (i) Summing all edits; or (ii) Weighing the edits by the proportional contribution of each v∈Va. Team composition features (n=5) The way a team of Wikipedians engages with one another and with the edited article is crucial for the team’s success and resilience (66). We hypothesize that this is also correlated with sustainable success. This feature set, named TC, includes collaborators-gini (TC1) and ip-based-edits-percentage (TC2). TC1 corresponds to the Gini inequality coefficient (85) among the Wikipedian collaborators. It is calculated in the same way as for discussers-gini (see previous paragraph) while taking into account article edit distribution among Wikipedians. TC2 represents the fraction of revisions done by anonymous editors. TC also includes features representing the relation between two populations: collaborators of the main article (Va) and discussers of the talk page (DIa). We create the following: fraction of main article collaborators who also continue to the talk page discussion (TC3), fraction of collaborators of the talk page who also contribute to the main article (TC4), and distribution-diff (TC5). TC5 captures how different is the contribution between editing and discussing among Wikipedians. The contribution is calculated as the proportional edits (or comments) over all revisions (or discussions). We first calculate the contribution difference between edits and discussions per a∈Va∪DIa and then average the differences. Supplementary Material