OpenChallenges: A FAIR-aligned, centralized platform for crowdsourced challenges in biomedical research Authors: Verena Chung, Rongrong Chai, Hallie Swan, Maria Diaz, Shysta Bynum, Adam Hindman, Jacob Albrecht, Serghei Mangul, Susheel Varma, Jay Hodgson, Solveig K. Sieberts, Alberto Pepe, Luca Foschini, Thomas Schaffter, Gaia Andreoletti Published: 2026 ABSTRACT SummaryOpen community challenges are critical for benchmarking computational methods in biomedical research, but the landscape is highly fragmented. Currently, over 2,000 challenges exist across a large and fragmented set of platforms, yet no centralized registry exists to index them. To address this, we developed OpenChallenges, a centralized, FAIR (Findable, Accessible, Interoperable, and Reusable)-aligned hub that aggregates and standardizes challenge metadata from diverse sources into a single, searchable registry. By implementing an updated metadata model explicitly mapped to FAIR dimensions, the platform advances challenge discoverability. A metadata completeness scoring framework assigns each challenge one of four tiers (baseline, standard, comprehensive, and gold standard), providing users with a transparent measure of metadata richness and guiding community curation priorities. Built on an open-source, API-driven architecture, OpenChallenges streamlines discovery, enhances visibility, and supports community-driven curation. This resource transforms a fragmented ecosystem into an interconnected knowledge base, advancing transparency in computational biology. Introduction Community challenges have emerged as a cornerstone of innovation in biomedical research. By uniting computational biologists, clinicians, statisticians, and data scientists, they catalyze cross-disciplinary collaborations essential to developing robust benchmarks and clinically actionable tools. These structured competitions have become indispensable instruments for rigorously evaluating computational methods on shared datasets, accelerating the pace of discovery in complex biological systems. The utility of this community-driven model is well documented by a legacy of pioneering initiatives. The Critical Assessment of Protein Structure Prediction (CASP),1,2 a long-running, community-wide blind challenge, has revolutionized structural bioinformatics by rigorously evaluating methods on unpublished experimental targets, setting the template for credible benchmarking and catalyzing breakthroughs such as deep-learning-based predictors. Similarly, the Dialogue for Reverse Engineering Assessment and Methods (DREAM) initiative has pioneered collaborative approaches, widening its scope from gene network reverse engineering to a broad array of biomedical challenges, such as predicting drug responses.3,4 Concurrently, the Critical Assessment of Genome Interpretation (CAGI) has systematically evaluated variant interpretation methods, providing essential insights into the reliability of computational genomics pipelines.5,6 The Critical Assessment of Functional Annotation (CAFA) is an ongoing, global, community-driven effort to evaluate and improve the computational annotation of protein function.7 Other long-standing efforts, including the Genetic Analysis Workshop (GAW),8 Critical Assessment of Massive Data Analysis (CAMDA),9 and challenges in medical imaging (e.g., MICCAI),10 have expanded this rigorous benchmarking into vital areas such as multi-omics integration and translational medicine. The impact of these challenges extends far beyond establishing state-of-the-art benchmarks. They shed light on new and existing datasets, evaluation metrics, and reproducible workflows that outlive the competitions themselves.11 By inviting broad participation, challenges democratize access to valuable, often private, biomedical data, lowering the barrier to entry for new researchers. This, in turn, stimulates methodological innovation and provides critical, hands-on training opportunities for early-career scientists, equipping them to handle complex biological datasets.12 However, the current biomedical challenge landscape is critically fragmented, which severely limits its collective potential. More than 2,000 challenges are scattered across over 26 distinct platforms, including Synapse, Kaggle, Grand Challenge, and precisionFDA, with many others run independently on GitHub or self-hosted websites and only a fraction of which are indexed in any centralized registry. This dispersion creates significant barriers. For participants, it hinders discoverability, making it difficult to find relevant challenges and limiting participation. For organizers, it complicates efforts to build on previous challenge designs and attract a diverse pool of participants. For the scientific community at large, it obstructs the long-term utility and analysis of benchmarking results, making systematic meta-analysis of methods nearly impossible. A lack of standardization compounds this fragmentation. Each platform employs its own data formats, metadata structures, and evaluation schemes.13 This inconsistency impedes cross-comparison of methods and makes reproducibility difficult to achieve, echoing a broader crisis in which a majority of researchers report failed attempts to reproduce published results.14 Consequently, the challenge ecosystem frequently fails to meet the FAIR (Findable, Accessible, Interoperable, and Reusable) data principles.15 The vital scientific resources generated, such as the datasets, results, and software, are often not findable due to the lack of a central registry. Their long-term accessibility is jeopardized when platforms become inactive or when datasets are retired. The lack of common metadata standards means they are likely not interoperable. As a result, these valuable resources are rarely reusable for future studies, leading to methodological redundancy and a loss of scientific knowledge. To improve the findability of challenges and catalog their FAIRness, we developed OpenChallenges (OC), a centralized, open-source, and FAIR-aligned hub that standardizes challenge metadata using a unified data model including biomedical ontologies. This enables powerful search across the full registry. It also rates each challenge with a metadata completeness score to quantify the FAIRness of challenge resources. The platform is designed not merely as a static catalog but as a living knowledge base, ensuring challenges and their outcomes remain discoverable and reusable long after completion. OpenChallenges is freely available at http://openchallenges.io/. Results Navigating the OC ecosystem OC provides a centralized, searchable challenge registry accessible via a clean top-level navigation menu. The registry can be filtered via a sidebar that offers a range of metadata attributes, including status, keywords, platform, input data type, incentive, submission type, and organizing community (Figure 1). An integrated chart panel displays the distribution of challenges by year and organizing community, giving users an at-a-glance overview of the registry’s composition before they begin filtering.Figure 1The OpenChallenges challenge registry interfaceThe filterable sidebar (left) allows users to narrow challenges by status, platform, keywords, input data type, evaluation metric, incentive, and organizing community. Each challenge card displays key metadata fields and a metadata completeness tier badge. The chart (top right) summarizes registry composition by year and organizing community. The OpenChallenges challenge registry interface The filterable sidebar (left) allows users to narrow challenges by status, platform, keywords, input data type, evaluation metric, incentive, and organizing community. Each challenge card displays key metadata fields and a metadata completeness tier badge. The chart (top right) summarizes registry composition by year and organizing community. Each challenge is presented as a card displaying its title, organizing community (if applicable), a description excerpt, status, start and end dates, platform, keywords (using standardized terms), incentive type, and metadata completeness score tier (described in the methods). The completeness tier is shown prominently on each card, providing users with an immediate signal of metadata richness before selecting an entry. Selecting a card directs the user to the challenge website, where users can learn more about the challenge, download data (if available), and participate if the challenge is still active. For a concrete example, a computational biologist seeking active genomics challenges can filter by input data type, status (submissions open), and incentive (publication opportunity), returning a curated list of relevant entries with direct links to challenge websites, consolidating discovery that would otherwise require searching across dozens of independent platforms. Growth and diversity of challenges within the OC registry The OC registry16 currently indexes 836 biomedical challenges aggregated from 26 distinct platforms, with an additional 75 challenges from smaller or miscellaneous sources grouped under an “other” category, spanning a broad range of domains, modalities, and task types (Table S1). The distribution reflects a landscape dominated by three major hubs including Grand Challenge (241 challenges, 28.8%), Synapse (190, 22.7%), and Kaggle (83, 9.9%), which together account for 61.5% of indexed challenges, alongside a substantial long tail of 38.5% (n = 322) from 23 additional platforms including EMBL-EBI, DrivenData, AIcrowd, CodaBench, and precisionFDA (Table S1). The registry spans challenges in 12 biomedical domains (including oncology, genetics and genomics, infectious disease, and neurology), 7 modalities (including radiology, omics, and natural language), and 7 task types (including classification, segmentation, drug response, and causal inference), organized through a controlled keyword vocabulary. Challenge creation has accelerated markedly in recent years, with 80 challenges launched in 2025, 68 in 2024, and 69 in 2023 and 133 challenges currently active or upcoming across the registry (Figure 2).Figure 2Growth of the OpenChallenges registry (2000–2025)(A) Cumulative number of indexed challenges by start year, stratified by platform type: major platforms (Synapse, Kaggle, and Grand Challenge; red), long-tail and independent platforms (blue), and all platforms combined (dashed black).(B) Annual new challenges added per year, shown as a stacked bar chart by platform type. The shaded region highlights the post-2020 period of accelerated growth. Note: 272 of 836 indexed challenges have no recorded start date and are excluded from both images. Growth of the OpenChallenges registry (2000–2025) (A) Cumulative number of indexed challenges by start year, stratified by platform type: major platforms (Synapse, Kaggle, and Grand Challenge; red), long-tail and independent platforms (blue), and all platforms combined (dashed black). (B) Annual new challenges added per year, shown as a stacked bar chart by platform type. The shaded region highlights the post-2020 period of accelerated growth. Note: 272 of 836 indexed challenges have no recorded start date and are excluded from both images. To quantify the scientific footprint of the indexed challenges, we enriched the subset of 111 challenges (13.3% of the registry) whose metadata include a DOI or preprint identifier using the OpenAlex open bibliographic graph. Of these, 106 resolved to indexed publications (95.5% resolution), which have collectively accumulated 28,274 citations (mean: 267, median: 85, maximum: 3,227). The most highly cited indexed challenges span medical imaging (CAMELYON16: 3,227 citations17; LUNA16: 1,29118), gene-network inference (DREAM5: 1,75519), protein interaction mapping (DREAM4 yeast interactome: 1,42220), and multi-task segmentation (Medical Segmentation Decathlon: 1,14121). Cross-validation against Semantic Scholar showed agreement within 20% on all top-10 challenges. Because only 13.3% of indexed challenges carry a persistent identifier, these figures represent a conservative lower bound on the total scientific output enabled by the community challenges indexed in OC; expanding persistent-identifier coverage is an active curation priority (Tables 1 and S2).Table 1Top 10 most-cited OpenChallenges entries by associated publicationRankChallenge/publication titleOrganizing communityYearCitations (OpenAlex)References1Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast CancerISBI20173,227Ehteshami Bejnordi et al.172Wisdom of crowds for robust gene network inferenceDREAM20121,755Marbach et al.193Large-Scale Mapping and Validation of Escherichia coli Transcriptional Regulation from a Compendium of Expression ProfilDREAM20071,633Faith et al.224High-Quality Binary Protein Interaction Map of the Yeast Interactome NetworkDREAM20081,422Yu et al.235Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomograph–20171,291Setio et al.186The Medical Segmentation Decathlon–20221,141Antonelli et al.217Comparison and Evaluation of Methods for Liver Segmentation From CT DatasetsMICCAI20091,116Heimann et al.248Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challengeMICCAI2013799Litjens et al.259Revealing strengths and weaknesses of methods for gene network inferenceDREAM2010787Marbach et al.2010CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentationISBI2020752Kavur et al.26Citation counts from OpenAlex (accessed April 2026). Cross-validation against Semantic Scholar showed agreement within 20% for all 10 entries (mean absolute discrepancy: 6.1%). Full per-challenge data for all 106 resolved publications is provided in Table S2. Table 1Top 10 most-cited OpenChallenges entries by associated publicationRankChallenge/publication titleOrganizing communityYearCitations (OpenAlex)References1Diagnostic Assessment of Deep Learning Algorithms for Detection of Lymph Node Metastases in Women With Breast CancerISBI20173,227Ehteshami Bejnordi et al.172Wisdom of crowds for robust gene network inferenceDREAM20121,755Marbach et al.193Large-Scale Mapping and Validation of Escherichia coli Transcriptional Regulation from a Compendium of Expression ProfilDREAM20071,633Faith et al.224High-Quality Binary Protein Interaction Map of the Yeast Interactome NetworkDREAM20081,422Yu et al.235Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomograph–20171,291Setio et al.186The Medical Segmentation Decathlon–20221,141Antonelli et al.217Comparison and Evaluation of Methods for Liver Segmentation From CT DatasetsMICCAI20091,116Heimann et al.248Evaluation of prostate segmentation algorithms for MRI: The PROMISE12 challengeMICCAI2013799Litjens et al.259Revealing strengths and weaknesses of methods for gene network inferenceDREAM2010787Marbach et al.2010CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentationISBI2020752Kavur et al.26Citation counts from OpenAlex (accessed April 2026). Cross-validation against Semantic Scholar showed agreement within 20% for all 10 entries (mean absolute discrepancy: 6.1%). Full per-challenge data for all 106 resolved publications is provided in Table S2. Top 10 most-cited OpenChallenges entries by associated publication Citation counts from OpenAlex (accessed April 2026). Cross-validation against Semantic Scholar showed agreement within 20% for all 10 entries (mean absolute discrepancy: 6.1%). Full per-challenge data for all 106 resolved publications is provided in Table S2. Metadata completeness analysis across all 836 challenges reveals substantial variation in documentation quality, reflecting the heterogeneity of source platforms. Overall, 7.5% of challenges (n = 63) achieved the comprehensive tier and 1.6% (n = 13) achieved gold standard, including all eight CACHE drug discovery challenges and both BraTS 2024 challenge series, which populated all 17 metadata fields including dataFormat, dataLicense, evaluationMetric, and submissionLicense (Figure 3). The majority of challenges fall into the standard (51.7%) or baseline (39.2%) tiers, with mean FAIR dimension scores of 80.0% (Findable), 50.2% (Accessible), 45.7% (Interoperable), and 11.0% (Reusable) (Figure 4). The consistently low reusable score reflects the sparse populations of evaluationMetric (6.5% coverage), submissionLicense (14.6%), and dataLicense (11.8%) across the registry, which are precisely the fields most critical for methodological reproducibility and cross-challenge benchmarking and the primary targets of ongoing community curation efforts.Figure 3Metadata completeness tier distribution of the OpenChallenges registry(A) Overall distribution of the 836 indexed challenges across four completeness tiers.(B) Tier distribution stratified by platform type, showing the proportion of challenges from major platforms (Synapse, Kaggle, and Grand Challenge; n = 514) and long-tail/independent platforms (n = 322) in each tier. Long-tail platforms show a higher proportion of baseline-tier entries, reflecting the sparser metadata natively available from smaller hosting environments.Figure 4FAIR dimension score profile of the OpenChallenges registry (N = 836)Radar chart showing mean scores for each of the four FAIR dimensions, findable (80.0%), accessible (50.2%), interoperable (45.7%), and reusable (11.0%), across all indexed challenges. Each dimension is scored as the mean proportion of its assigned metadata fields that are populated. The shaded band represents ±1 standard deviation. Metadata completeness tier distribution of the OpenChallenges registry (A) Overall distribution of the 836 indexed challenges across four completeness tiers. (B) Tier distribution stratified by platform type, showing the proportion of challenges from major platforms (Synapse, Kaggle, and Grand Challenge; n = 514) and long-tail/independent platforms (n = 322) in each tier. Long-tail platforms show a higher proportion of baseline-tier entries, reflecting the sparser metadata natively available from smaller hosting environments. FAIR dimension score profile of the OpenChallenges registry (N = 836) Radar chart showing mean scores for each of the four FAIR dimensions, findable (80.0%), accessible (50.2%), interoperable (45.7%), and reusable (11.0%), across all indexed challenges. Each dimension is scored as the mean proportion of its assigned metadata fields that are populated. The shaded band represents ±1 standard deviation. Minimal information about a challenge schema cross-domain validation The minimal information about a challenge (MIAC) schema was derived from an exploratory analysis of 47 DREAM challenges (Synapse.org, 2012–2021), yielding 50 standardized parameters across five categories (Table S3). An empirical validation, through a gap analysis of 30 non-DREAM challenges across three domain categories, confirmed cross-domain applicability: core fields (title, platform, status, and description) achieved 93%–100% coverage across all domains, with an overall mean of 61.2% (Table S4). Fields with the lowest cross-domain coverage were evaluationMetric and submissionLicense (both 7% overall) and doi (7% overall), confirming that the completeness scoring framework addresses a genuine and consistent gap across the challenge landscape rather than one specific to DREAM challenges. Community curation of OC OC is designed as a living registry, continually improved by the global community of researchers and challenge organizers rather than a centralized editorial team. The platform’s source code is fully open source and, by design, no single entity owns the aggregated metadata. Community members can contribute or update challenge entries directly via the OC discussion forum (https://www.synapse.org/Synapse:syn51476216/discussion/) or the Challenges and Benchmarking Service Desk (sagebionetworks.jira.com/servicedesk/customer/portal/18), without requiring platform redeployment. This distributed curation model is central to the platform’s long-term sustainability and its ability to scale with the growth of the biomedical challenge landscape. Discussion OC represents a significant advancement toward establishing a more cohesive, FAIR-aligned challenge ecosystem in biomedical research. More broadly, we argue that community challenges should be treated as shared scientific infrastructure: public goods whose datasets, evaluation protocols, and benchmark results warrant the same persistence, stewardship, and discoverability standards that the field has established for reference genomes, ontologies, and data commons. Framing challenges this way reorients the conversation from individual competitions toward cumulative, comparable, and reusable benchmarking as a foundation for computational biomedicine. While individual hosting platforms have traditionally operated in isolation, creating fragmented knowledge silos, OC provides a comprehensive meta-infrastructure that systematically consolidates, annotates, and preserves the outputs of disparate computational challenges. A standardized schema is central to this effort, ensuring the consistent capture of core descriptors, including datasets, evaluation metrics, and organizing institutions. This standardized schema builds upon precedents set by related frameworks, such as the OpenEBench benchmarking data model27 and the BIAS model for medical imaging.28 However, while they share these conceptual foundations, OpenEBench and OC serve distinct roles in the broader ecosystem. OpenEBench goes deep, providing the active infrastructure to run, evaluate, and store specific bioinformatics benchmarks. In contrast, OC goes broad, acting as a cross-domain registry that aggregates metadata to make challenges highly findable without executing the benchmarks itself. The OC open architecture also directly addresses sustainability concerns, as project-specific platforms may lose support over time, as illustrated by Nightingale OS, which is no longer an active challenge-hosting platform (although the data are still available). By providing persistent infrastructure, OC ensures that datasets, evaluation frameworks, and outcomes remain accessible and reusable even after their original host is no longer maintained. Complementing these platform-level efforts, the recently established Benchmarking Consortium for AI in Healthcare (BEACON; https://conscience.ca/beacon/) brings together benchmarking initiatives with the explicit aim of addressing fragmentation and improving coordination across the field. Because OC shares this motivation at the metadata and discoverability layer, we view BEACON as a natural stakeholder in the long-term sustainability and coordination of the registry and anticipate that closer alignment with such consortia will further strengthen the cross-platform challenge ecosystem. Despite its comprehensive design, OC has limitations. First, not all challenge platforms provide standardized APIs, necessitating manual curation workflows that can cause delays and incomplete coverage. Second, the challenges currently indexed may not represent the complete landscape of biomedical challenges, as our hybrid collection methods may miss smaller, independent, or less publicized competitions. As the platform’s collection grows, it will become an increasingly valuable resource for analyzing the full spectrum of community benchmarking activities. It is important to note, however, that the value of a centralized registry is arguably greatest for challenges hosted outside major platforms. Of the 836 challenges currently indexed, 38.5% (n = 322) originate from 22 long-tail and independent sources, including EMBL-EBI, DrivenData, AIcrowd, CodaBench, and precisionFDA, which might lack the inherent discoverability of larger hubs such as Synapse, Kaggle, and Grand Challenge. Furthermore, even within these major hubs, the registry provides crucial standardization. Platforms such as Synapse often do not enforce a uniform layout or structure for their challenge pages. Consequently, essential metadata is frequently scattered across disparate wiki formats, making programmatic curation and automated web scraping exceedingly difficult. Metadata completeness analysis further supports this: challenges from long-tail platforms have a lower mean completeness score compared to those from major platforms, reflecting the sparser metadata these sources provide natively (Figure 3). By indexing these challenges within a standardized schema and surfacing their completeness tier, OC provides the greatest incremental discoverability benefit precisely for the challenges that need it most. To maintain comprehensive registry coverage as the challenge landscape grows, OC employs a multi-pronged approach. Programmatic ingestion and AI-assisted extraction provide a scalable foundation, while structured manual curation addresses platforms lacking automated access. Ultimately, community-driven contributions ensure that these aggregated metadata are collaboratively validated and enriched. Planned future work includes targeted outreach to challenge organizers in underrepresented domains and integration of additional ontologies to improve semantic discoverability for niche research areas. Third, OC does not currently encompass private, industry-led challenges, which restricts its comprehensiveness. Fourth, metadata drift remains an inherent risk as source platforms evolve or retire. Finally, our current ontology integration, while functional, is limited in scope. These are challenges we are actively working to address, underpinned by a long-term institutional commitment to the platform. Beyond architectural openness, Sage Bionetworks commits to hosting OC as durable public infrastructure within the Synapse ecosystem, where it benefits from the same operational, security, and continuity guarantees as other Sage-maintained data portals. The cost of maintaining the registry is low and decreasing: the ingestion pipeline combines platform APIs, AI-assisted extraction, and increasingly research workflows that reduce human curation to targeted review and validation rather than primary data entry. This cost structure makes long-term stewardship tractable even as the registry grows. Governance could be further strengthened by a transition toward direct community contribution: beyond the existing discussion forum and Service Desk channels, we are designing lightweight in-platform mechanisms through which challenge organizers can submit, claim, and curate their own entries, with provenance tracking and optional expert review. Together, institutional hosting, AI-augmented maintenance, and direct community contribution form a three-layered sustainability model designed to keep OC current without dependency on any single funding cycle or curation team. Relatedly, metadata completeness varies across indexed challenges, reflecting the heterogeneity of information available from source platforms, a limitation inherent to any aggregated registry operating at this scale. Fields such as dataLicense, submissionLicense, and evaluationMetric are particularly difficult to retrieve programmatically because they are frequently absent from platform APIs and require either manual curation or direct input from challenge organizers. To address this systematically, we have implemented a metadata completeness scoring framework that assigns each challenge a tier from baseline (meaning it needs curation) to comprehensive/gold standard, displayed transparently on each challenge card. We anticipate that this visibility will encourage challenge organizers to contribute missing metadata directly, progressively improving registry quality over time. Enriching these fields remains an active curation priority. The low persistent-identifier coverage observed in the registry (13.3% of challenges carry a DOI or arXiv link) is itself a symptom of the fragmentation OC aims to address; the scientific footprint analysis of the DOI-indexed subset—over 28,000 citations across 106 resolved publications—illustrates both the substantive value of the underlying work and the magnitude of the discoverability gap for the remaining 87% of challenges that lack persistent identifiers. While the technical architecture of OC provides the foundation for a sustainable registry, long-term success depends equally on sustained community participation. Although initial curation has been led by Sage Bionetworks, the vision is for the platform to evolve into a fully community-driven resource, and transitioning to this crowdsourced model is central to its ability to scale with the rapidly expanding biomedical challenge landscape. We have therefore designed complementary mechanisms to encourage challenge organizers to register and maintain their entries. First, each OC entry provides persistent, citable attribution to the hosting community (such as DREAM and MICCAI), creating a stable record of benchmarking contributions that can be referenced in publications and grant applications, fulfilling a role analogous to data repository accession numbers in genomics. Second, the metadata completeness scoring framework directly rewards contribution effort: challenges that achieve the comprehensive or gold-standard tier gain greater visibility through semantic search and filtered discovery, creating a measurable return on the investment of enriching metadata. Third, we will be exploring integration with publication workflows, wherein journals could recommend or require OC registration as part of manuscript submission for challenge-based studies, analogous to existing mandates for genomic data deposition. Fourth, alignment with open-science mandates provides institutional-level incentives for publicly funded challenge organizers: the NIH Data Management and Sharing Policy (NOT-OD-21-013, effective January 202329) requires persistent, FAIR-aligned sharing of scientific data and associated metadata, and ELIXIR’s FAIR data guidelines impose analogous expectations across European bioinformatics infrastructure, positioning an open, FAIR-aligned registry such as OC as a natural compliance and discovery layer for challenge-generated resources. Finally, OC can provide organizers with quantitative insights into challenge visibility and participation trends (derived from platform analytics), offering a practical utility that rewards ongoing engagement beyond initial registration. We recognize that fully realizing these mechanisms will require sustained partnership with journals, funders, and hosting platforms, and we view incentive design as a critical area of ongoing development alongside the platform’s technical roadmap. Realizing the full potential of a cross-platform foundation and NIH-supported data commons could substantially broaden both coverage and impact. We view partnerships with scientific journals as particularly critical: journals are uniquely positioned to make registration of challenge metadata a standard expectation at submission or publication, analogous to existing mandates for genomic data deposition, sequence accession, and code availability. Such integration would transform OC from an opt-in registry into a default component of the benchmarking publication workflow, closing the loop between challenge execution, scientific reporting, and long-term discoverability. We invite journals, funders, and hosting platforms to engage with us in shaping this integration. Several promising avenues for enhancement emerge from this foundational work. We plan to expand semantic enrichment by integrating additional ontologies (e.g., SNOMED CT30) to improve search precision. Ongoing validation of the OC schema against emerging challenge types, including foundation model evaluations, federated learning benchmarks, and multi-modal challenges, will be important as the biomedical benchmarking landscape continues to evolve. The gap analysis presented in Table S4 provides a baseline for this effort, identifying evaluationMetric, submissionLicense, and doi as the fields most in need of community curation and schema refinement for underrepresented challenge types. We will also explore federated challenge networks that enable distributed experiments while maintaining centralized metadata coordination. Furthermore, implementing AI-driven analytics on the aggregated metadata could identify emerging research trends and methodological gaps. By consolidating a fragmented landscape into a FAIR-aligned, community-driven registry, OC lays the groundwork for sustainable benchmarking infrastructure that strengthens the role of challenges as engines of discovery in computational biology. Methods Metadata acquisition and curation Challenge metadata were collected from a broad range of public hosting platforms, including Grand Challenge, Synapse, Kaggle, precisionFDA, and DrivenData. The acquisition method varied by platform and followed a three-tier approach. Where available, metadata were retrieved programmatically via platform-specific APIs or bulk data downloads. For platforms with standardized web templates but lacking direct API access, metadata were extracted using AI-assisted methods, leveraging the Gemini Co-Scientist to systematically parse and structure challenge information at scale. For platforms lacking both automated access and standardized templates, metadata were obtained through structured manual curation of challenge websites, publications, and repositories. Community-driven contributions via the Synapse discussion forum provide an additional channel through which challenge organizers and domain experts can directly submit, validate, and enrich metadata entries. This hybrid approach ensures broad coverage across both legacy and ongoing challenges, including those hosted on smaller or independent platforms. The resulting curated metadata are deposited as a Synapse data record.16 Development and refinement of the MIAC schema The MIAC schema was developed as the foundation for the OC data model. The schema’s design originated from an exploratory analysis of 47 DREAM challenges (Synapse.org, 2012–2021), which yielded 50 standardized parameters across five categories: challenge, organizing group, subtasks, data, and results (Table S3). To ensure interoperability and generalizability across diverse domains, the initial schema underwent iterative cross-domain refinement through systematic comparison with two independent frameworks: the BIAS model,31 used by Grand Challenge for medical imaging benchmarks, and the OpenEBench Benchmarking Data Model,13 developed under the ELIXIR EXCELERATE initiative for bioinformatics tool evaluation. This triangulation across three distinct challenge ecosystems, community biology (DREAM), medical imaging (Grand Challenge), and bioinformatics tools (OpenEBench), was designed to ensure that the resulting schema captured metadata elements common across domains rather than specific to any single platform or field. To provide empirical evidence of cross-domain applicability, we conducted a formal gap analysis evaluating MIAC field coverage across a stratified sample of 30 non-DREAM challenges drawn from three domain categories: medical imaging (n = 10, Grand Challenge), general ML/biology (n = 10, including Kaggle, CodaBench, AIcrowd, DrivenData, and EvalAI), and niche or independent platforms (n = 10) (Table S4). The final MIAC schema served as the foundational blueprint for the OC data models. Each MIAC parameter was mapped directly to attributes within the system’s architecture. For example, the MIAC “challenge name” parameter became a primary field in the Challenge data model, which now serves as a key mechanism for searching challenges on the OC platform. These data models, in turn, dictated the schema and data types for the underlying database tables, ensuring structured and consistent storage of all challenge and organization information. Data transformation and harmonization Following acquisition, raw metadata were normalized and harmonized into the standardized OC data model. Each record was mapped to a defined set of fields including challenge title, hosting community, description, start and end dates, submission format, evaluation metrics, and associated identifiers (DOIs and resource URLs) to preserve provenance. To ensure programmatic consistency across the registry, internal controlled vocabularies were enforced for categorical fields, including challenge status, data license, access type, incentives, submission types, and evaluation metrics. Furthermore, external ontologies were applied to enrich the metadata semantically: EDAM32 was used for computational concepts and ICD-1033 for clinical entities, ensuring challenges are discoverable using standard biomedical terminology. Validation and quality control Quality control was performed at multiple stages. Automated scripts verified data completeness, type consistency, and adherence to the schema. Key validation rules included (1) non-null identifiers for challenges and organizations, (2) valid date ranges, and (3) consistency between platform metadata and imported fields. Entries flagged for errors were subject to manual curation by domain experts to ensure fidelity. This hybrid validation framework minimized both systematic errors and metadata drift across updates. Metadata completeness analysis To quantify the FAIRness of individual challenge records within the registry, we implemented a metadata completeness scoring framework for the OC data model. For each challenge, a completeness score was calculated as the proportion of populated fields out of the 17 fields defined in the updated OC data model: name, description, website_url, keywords, doi, input_data_type, data_format, data_license, data_access_type, platform, start_date, end_date, status, incentive, submission_type, submission_license, and evaluation_metric (Table S5). Based on this score, each challenge was assigned to one of four tiers: baseline (0%–49%), standard (50%–59%), comprehensive (60%–79%), or gold standard (80%–100%). These tiers were designed to reflect meaningful distinctions in metadata richness, from records requiring substantial curation effort to those providing sufficient detail for discovery, participation, and reuse in line with FAIR principles. Completeness scores are computed automatically at indexing time and are updated dynamically as new metadata are contributed by the community or retrieved from source platforms. Web application and programmatic access OC has been rebuilt into the Synapse.org ecosystem, transitioning from a standalone platform to a dedicated Challenges Portal hosted within Synapse’s infrastructure. This migration consolidates the registry within a mature, actively maintained open-science platform, ensuring long-term sustainability and reducing the operational overhead of maintaining independent infrastructure. The Challenges Portal is built on the synapse-web-monorepo,34 an open-source TypeScript repository maintained by Sage Bionetworks that powers all Synapse-based data portals. The portal uses the synapse-react-client component library for challenge browsing, filtering, and metadata display and is continuously deployed via GitHub Actions with separate staging and production environments. All challenge metadata are stored in Synapse Tables, providing structured, SQL-queryable, versioned storage with fine-grained access controls. Metadata can be accessed and updated programmatically via the Python Synapse client (synapseclient, available via PyPI) or through the Synapse web interface, enabling researchers, curators, and challenge organizers to interact with the registry without requiring platform redeployment. Authentication and access control are managed through existing Synapse user accounts, eliminating the need for a separate credentialing system. Comprehensive documentation for the Synapse platform is available at https://docs.synapse.org/, including information on challenges, API clients (Python, R, CLI, and REST), and other platform capabilities. Resource availability Lead contact Requests for further information and resources should be directed to and will be fulfilled by the lead contact, Gaia Andreoletti (gaia.andreoletti@sagebase.org). Materials availability This study did not generate new materials. Data and code availability •The OC curated challenge metadata registry has been deposited at Synapse and is publicly available as of the date of publication at https://doi.org/10.7303/syn71324868. This paper analyzes existing, publicly available data, which are accessible at the source hosting platforms cataloged in Table S1 and indexed within the registry. The underlying datasets powering the portal are updated daily.•All original code has been deposited at Synapse and is publicly available as of the date of publication at https://doi.org/10.7303/syn75066676 (synapse-web-monorepo, OC challenges application at apps/portals/challenges). Active development continues on GitHub.•Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request. •The OC curated challenge metadata registry has been deposited at Synapse and is publicly available as of the date of publication at https://doi.org/10.7303/syn71324868. This paper analyzes existing, publicly available data, which are accessible at the source hosting platforms cataloged in Table S1 and indexed within the registry. The underlying datasets powering the portal are updated daily. The OC curated challenge metadata registry has been deposited at Synapse and is publicly available as of the date of publication at https://doi.org/10.7303/syn71324868. This paper analyzes existing, publicly available data, which are accessible at the source hosting platforms cataloged in Table S1 and indexed within the registry. The underlying datasets powering the portal are updated daily. •All original code has been deposited at Synapse and is publicly available as of the date of publication at https://doi.org/10.7303/syn75066676 (synapse-web-monorepo, OC challenges application at apps/portals/challenges). Active development continues on GitHub. All original code has been deposited at Synapse and is publicly available as of the date of publication at https://doi.org/10.7303/syn75066676 (synapse-web-monorepo, OC challenges application at apps/portals/challenges). Active development continues on GitHub. •Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request. Acknowledgments We thank the OC contributors and the broader community of challenge organizers and participants for sharing metadata, feedback, and curation efforts that made this registry possible. This work was supported by the 10.13039/100000054National Cancer Institute (NCI) under award number U24CA248265. Author contributions Conceptualization, T.S., J.A., L.F., S.V., and G.A.; methodology, V.C., R.C., G.A., and T.S.; software, V.C., R.C., H.S., M.D., S.B., A.H., S.K.S., A.P., J.A., and J.H.; investigation and data curation, V.C., R.C., M.D., and G.A.; validation, V.C., R.C., and G.A.; writing – original draft, G.A., T.S., and S.M.; writing – review & editing, all authors; visualization, S.B., A.H., V.C., R.C., and G.A.; supervision, T.S. and G.A.; funding acquisition, J.A., S.V., and L.F.; project administration, G.A. Declaration of interests The authors declare no competing interests. Declaration of generative AI and AI-assisted technologies in the writing process During the preparation of this work, the authors used Claude Opus 4.7 (Anthropic) to polish the language. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.