An Evaluation Framework for the FAIR Assessment tools in Open Science
Payel Patra, Daniele Di Pompeo, Antinisca Di Marco · 2025
Read An Evaluation Framework for the FAIR Assessment tools in Open Science by Payel Patra, Daniele Di Pompeo, Antinisca Di Marco (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-15929.txt.
Introduction
Payel Patra, Daniele Di Pompeo, and Antinisca Di Marco build a framework for judging the judges of FAIR research data. Their work appeared in twenty twenty-five. FAIR data are findable, accessible, interoperable, and reusable, and many tools now score how well platforms meet those aims.
The authors evaluate twenty-two such assessment tools against open science platforms. Their framework exposes where the tools agree, where they fall short, and where they waste effort. Better scoring, they argue, leads to better sharing and tidier research artifacts, from data to code and documentation. The study is aimed at platform builders who want honest measurement before claiming openness. It also seeds ideas for the next generation of platforms.
Catalog record
| CREATOR | Payel Patra, Daniele Di Pompeo, Antinisca Di Marco |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-15929
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Evaluation Framework for the FAIR Assessment tools in Open Science Payel Patra, Daniele Di Pompeo, Antinisca Di Marco Abstract Open science represents a transformative research approach essential for enhancing sustainability and impact. Data generation encompasses various methods, from automated processes to human-driven inputs, creating a rich and diverse landscape. Embracing the FAIR principles—making data and, in general, artifacts (such as code, configurations, documentation, etc) findable, accessible, interoperable, and reusable— ensures research integrity, transparency, and reproducibility, and researchers enhance the efficiency and efficacy of their endeavors, driving scientific innovation and the advancement of knowledge. Open Science Platforms OSP (i.e., technologies that publish data in a way that they are findable, accessible, interoperable, and reusable) are based on open science guidelines and encourage accessibility, cooperation, and transparency in scientific research. Evaluating OSP will yield sufficient data and artifacts to enable better sharing and arrangement, stimulating more investigation and the development of new platforms. In this paper, we propose an evaluation framework that results from evaluating twenty-two FAIR-a tools assessing the FAIR principles of OSP to identify differences, shortages, and possible efficiency improvements. 1 Introduction Open science aims to provide access to scientific research and its dissemination to all facets of society [ 1 , 2 ] . Research artifacts are among the most valuable resources in open science since they serve as the basis for innovations that result in societal advantages like developing alternative energy sources or disease treatments. All digital resources should be able to be found, accessed, interoperable, and reused by machines and humans, according to the FAIR Principles . For academics to publish scholarly results FAIRly must comply with the FAIR Principles [ 3 ] , which serve as guidelines for the kinds of practices researchers and scientists should demand more and more from digital resources. The primary strategy is to keep the FAIR standards up-to-date across the global repositories. To achieve this goal, several tools help scientists make their artifacts (e.g., data, knowledge, code, etc) public by respecting the FAIR principles. Since the FAIR principles are fundamental for Open Science [ 4 ] , several platforms helping the researcher to publish their results are coming into action (e.g., Zenodo , SoBigData , etc). In the following, we call such frameworks Open Science Platforms (OSP). Several FAIR assessment tools (called in the following FAIR-a tools) have been implemented to help scientists and researchers select the most appropriate OSP for their aims and scopes. Such tools assess how well the OSP frameworks implement the FAIR principles. The main goal of this paper is to study FAIR-a tools and provide a means to evaluate and classify them to help the open science community select the best FAIR-a tool to be used to assess to what extent their publishing strategy responds to the FAIR principles. Our aim is to identify FAIR-a tools differences, shortages, and possible efficiency improvements. In this paper we answer the following research questions: \circ RQ1: Which are the relevant features to be considered when assessing FAIR principles? \circ RQ2: Is the proposed evaluation framework suitable for classifying the existing FAIR-a tools? \circ RQ3: Which FAIR-a tool performs better considering user perception? What are the primary shortages, and what are the possible efficiency improvements? The contributions of this paper are: i) the specification of an evaluation framework for the FAIR-a tools useful to classify the existing FAIR-a tools. The evaluation framework has been defined using a bottom-up approach that extracts relevant features resulting from analyzing twenty-two FAIR-a tools linked at the webpage 1 1 1 https://fairassist.org/ ; ii) leveraging the defined evaluation framework, we provide an initial assessment of the existing FAIR-a tools; and finally, iii) we report the list of best tools considering the user perspective. In future work, we aim to enlarge the toolset to consider and involve more experts from science and industry in evaluating and improving the provided evaluation framework and its usage in the broader FAIR-a toolset. 2 Research Motivation and Objectives Numerous OSPs (that help open scientists to publish their dataset [ 5 ] and artifacts following the FAIR principles) and assessment techniques (that assess to what extent the OSP adheres to the FAIR principles) have been published in earlier research [ 7 ] . Due to the growing importance of open science, confirming the compliance of OSP to FAIR principles is now of paramount importance and requires a full investigation In particular, OSP aim to: \circ ensure data integrity and trust, maintain high-quality, reliable repositories that meet Findable and Accessible standards. Enhancing data accessibility boosts research impact and encourages broader use of research findings, adhering to the Accessible principle. \circ Enhance data management effectiveness and efficiency by identifying and strengthening key areas. Promote stakeholder collaboration by implementing standardized data management planning (DMP) procedures, thus improving Interoperable and Reusable data practices. Keeping these in mind, we can see how important it is to have FAIR assessment (FAIR-a) tools able to assess to what extent the OSP implements FAIR principles. However, FAIR-a tools currently have some shortcomings as well, therefore analyzing these tools might help identify the following points: \circ It makes choosing the best tool easier. \circ Direct development and improvement of Tool’s activities. \circ It aids in the standardization of FAIR data practices. \circ This will encourage developers to give FAIR principles priority. \circ It will Organize research processes more efficiently. \circ It Provides transparency into tool’s functionality and limitations. In [ 8 ] , N. Krans and the Karolinska Institutet research group (2022) evaluated ten FAIR assessment tools for chemicals and nanomaterials safety, noting inconsistencies in results and the need for consensus on assessment criteria. Tools like the ARDC FAIR self-assessment tool and FAIRshake offer various features, with SATIFYD recommended for quick assessments and automated tools aiding workflow development. Achieving consensus on FAIR criteria is vital for tool development. While online self-assessment tools suit individual datasets, (semi-) automated tools are better for entire databases. Compared to [ 8 ] , our study defines a novel evaluation framework with a bottom-up approach that evaluates 22 FAIR-a tools and provides detailed features and thorough assessment findings. Our comprehensive approach ensures superior performance and usability. The novelty and significance of our results within the FAIR context are underscored by the definition of an evaluation framework for FAIR-a tools, a detailed explanation of assessment results, and tool performance quality for these 22 tools. Moreover, this study goes beyond [ 8 ] because we evaluated 22 FAIR-a tools while [ 8 ] evaluated only 10 tools. This comprehensive assessment offers researchers a richer understanding and more precise differentiation of FAIR-a tools based on their qualities and capabilities, providing substantial insights for the research community. 3 Research Methodology Figure 1 sketches the methodology we followed in this study. It is composed by 8 steps starting with the selection of 22 FAIR-a tools publicly available. Figure 1: FAIR-a tools evaluation methodology Then, we conducted a thorough assessment of the considered toolset to ensure an effective evaluation process, yielding precise outcomes (step 2 in the figure). During the assessment, we note the dataset’s qualities (step 3), facilitating the identification and differentiation of relevant attributes (steps 4 and 5) that formed the foundation for developing a robust framework, significantly