NFDI4DS Shared Tasks for Scholarly Document Processing

Raia Abu Ahmad, Rana Abdulla, Tilahun Abedissa Taffa, Sören Auer, Hamed Babaei Giglou, Ekaterina Borisova, Zongxiong Chen, Stefan Dietze, Jennifer D’Souza, Mayra Elwes, Genet-Asefa Gesese, Shufan Jiang, Ekaterina Kutafina, Philipp Mayr, Georg Rehm, Sameer Sadruddin, Sonja Schimmler, Daniel Schneider · 2025

Read NFDI4DS Shared Tasks for Scholarly Document Processing by Raia Abu Ahmad, Rana Abdulla, Tilahun Abedissa Taffa, Sören Auer, Hamed Babaei Giglou, Ekaterina Borisova, Zongxiong Chen, Stefan Dietze, Jennifer D’Souza, Mayra Elwes, Genet-Asefa Gesese, Shufan Jiang, Ekaterina Kutafina, Philipp Mayr, Georg Rehm, Sameer Sadruddin, Sonja Schimmler, Daniel Schneider (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-22141.txt.

Introduction

Raia Abu Ahmad and twenty co-authors report on a set of shared tasks for processing scholarly documents. Shared tasks invite many research teams to solve the same problem under the same scoring rules, which moves a field forward in a comparable way. Their 2025 paper, accepted at a research data workshop, gives an updated overview of twelve such tasks run under Germany's national research data infrastructure for data science and AI.

The tasks cover a diverse range of challenges in handling scholarly documents, and they were hosted at leading venues in the field. Each task contributes open data sets, models, and tools for the wider community. Those outputs feed into the consortium's research data infrastructure. The authors frame shared tasks as engines for research that is findable, accessible, interoperable, and reusable, and for work that others can check and reproduce.

Catalog record

CREATORRaia Abu Ahmad, Rana Abdulla, Tilahun Abedissa Taffa, Sören Auer, Hamed Babaei Giglou, Ekaterina Borisova, Zongxiong Chen, Stefan Dietze, Jennifer D’Souza, Mayra Elwes, Genet-Asefa Gesese, Shufan Jiang, Ekaterina Kutafina, Philipp Mayr, Georg Rehm, Sameer Sadruddin, Sonja Schimmler, Daniel Schneider
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-22141

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

NFDI4DS Shared Tasks for Scholarly Document Processing Raia Abu Ahmad, Rana Abdulla, Tilahun Abedissa Taffa, Sören Auer, Hamed Babaei Giglou, Ekaterina Borisova, Zongxiong Chen, Stefan Dietze, Jennifer D’Souza, Mayra Elwes, Genet-Asefa Gesese, Shufan Jiang, Ekaterina Kutafina, Philipp Mayr, Georg Rehm, Sameer Sadruddin, Sonja Schimmler, Daniel Schneider, Kanishka Silva, Sharmila Upadhyaya, Ricardo Usbeck Abstract Shared tasks are powerful tools for advancing research through community-based standardised evaluation. As such, they play a key role in promoting findable, accessible, interoperable, and reusable (FAIR), as well as transparent and reproducible research practices. This paper presents an updated overview of twelve shared tasks developed and hosted under the German National Research Data Infrastructure for Data Science and Artificial Intelligence (NFDI4DS) consortium, covering a diverse set of challenges in scholarly document processing. Hosted at leading venues, the tasks foster methodological innovations and contribute open-access datasets, models, and tools for the broader research community, which are integrated into the consortium’s research data infrastructure. [email protected] [email protected] [email protected] [email protected] [email protected] [email protected] [email protected]@gesis.org [email protected] [email protected]@[email protected]@uni-koeln.de [email protected] [email protected] [email protected] [email protected]@uni-leipzig.de [email protected] [email protected] [email protected] keywords Shared Tasks NFDI4DS NFDI 1 Introduction Shared tasks are community-driven challenges that compare computational methods on specific problems using shared datasets and metrics [ Pa17 ] . They advance research by fostering innovation, encouraging collaboration, and establishing state-of-the-art methods across tasks and fields [ FU18 ] . Shared tasks offer multiple benefits to the community by promoting transparency and reproducibility, allowing for a fair comparison of methodologies, and resulting in publicly available resources (e. g., datasets, open-source software) [ Es21 ] . NFDI4DS (National Research Data Infrastructure for Data Science and Artificial Intelligence) is a German consortium that focuses on supporting the research data lifecycle in Data Science and Artificial Intelligence (AI) [ Sc23 ] . 1 1 1 https://www.nfdi4datascience.de This includes collecting, processing, analysing, publishing, and reusing artifacts such as datasets, models, code, and publications. NFDI4DS pays special attention to the FAIR principles as well as to transparency and reproducibility through open-source tools and rich documentation. Shared tasks play an important role in NFDI4DS, serving as mechanisms for community engagement and driving the development and refinement of tools and methods for its FAIR research data infrastructure. They also support NFDI’s broader mission of building a sustainable, interoperable, and accessible national research data infrastructure by promoting interdisciplinary collaboration, standardisation efforts, and long-term reuse of research outputs. To reach a broader audience and encourage widespread participation, NFDI4DS shared tasks are usually hosted at well-known venues. Currently, these include 1. the Natural Scientific Language Processing and Research Knowledge Graphs (NSLP) workshop series, initiated by NFDI4DS; 2 2 2 https://nfdi4ds.github.io/nslp2025/ and [ Re24 ] 2. the Scholarly Document Processing (SDP) workshop series [ Ch20 ] ; 3 3 3 https://sdproc.org 3. the Conference and Labs of the Evaluation Forum (CLEF); 4 4 4 https://clef2025.clef-initiative.eu/ 4. the Workshop on Semantic Evaluation (SemEval); 5 5 5 https://semeval.github.io and 5. the Int. Semantic Web Conference (ISWC) challenges. 6 6 6 https://iswc2025.semanticweb.org/#/program/challenges This paper provides an overview of shared tasks organised by NFDI4DS, extending [ Ab23 ] . The twelve tasks demonstrate how these competitions can drive methodological progress and contribute to the development of a FAIR research data infrastructure. 2 Completed Shared Tasks 2.1 SOMD 2025: Software Mention Detection Organiser: GESIS – Leibniz Institute for the Social Sciences Although software is integral to scientific discovery and analysis, it is rarely cited with the same consistency or formality as literature. Automatic identification and disambiguation of software mentions and related attributes (URLs, Versions, License, etc.) enhance accessibility and reproducibility but remain challenging [ Sc21 ] . SOMD 2025 [ Up25 ] is a shared task focused on extracting software mentions from scholarly articles. Following SOMD 2024 [ KKD24 ] , the task focused on joint learning of software mentions, attribute extraction, and classification of the their relations. The task was hosted at SDP 2025, 7 7 7 https://sdproc.org/2025/somd25.html and included two phases: Phase I involved model development using a gold standard training set [ Sc21 ] , evaluated using the average score of the entities extraction and relation classification tasks, while phase II tested generalisability on an out-of-distribution set. The competition (March/April 2025) attracted six teams, the best one achieved F1 scores of 0.89 (Phase I) and 0.63 (Phase II), surpassing baselines of 0.804 and 0.491, respectively. 2.2 SciVQA: Scientific Visual Question Answering Organiser: Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) The first SciVQA shared task [ BRR25 ] , part of SDP 2025, 8 8 8 https://sdproc.org/2025/scivqa.html is about developing question answering (QA) systems based on images of scientific figures, their captions, and metadata. The challenge aims to evaluate the recognition and reasoning capabilities of multimodal large language models (LLMs) on closed-ended visual vs. non-visual questions. The former refers to questions addressing visual attributes such as colour, shape, direction, height, size or position of objects, while the latter does not incorporate any of these aspects. The novel and large-scale SciVQA dataset 9 9 9 https://huggingface.co/datasets/katebor/SciVQA comprises 3000 images of figures from arXiv pre-prints and ACL Anthology publications. Each figure is semi-automatically annotated according to a custom QA schema featuring seven types, resulting in a total of 21000 English QA pairs. The benchmark contains several types of figures (e. g., line graph, bar chart, venn diagram, confusion matrix), also distinguishing between compound and non-compound figures. The evaluation is based on precision, recall and F1-scores of ROUGE-1, ROUGE-L [ Li04 ] , and BertScore [ Zh20 ] . 10 10 10 https://www.codabench.org/competitions/5904/ The competition attracted 20 registrations, with seven leaderboard submissions and five papers reporting their solutions. The highest-ranking team achieved scores of 0.80 for ROUGE-1 and ROUGE-L and 0.98 for BertScore. 2.3 ClimateCheck: Scientific Fact-checking of Social Media Posts on Climate Change Organiser: Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) ClimateCheck [ AUR25a ] was hosted at SDP 2025. 11 11 11 https://sdproc.org/2025/climatecheck.html The task addressed the growing challenge of misinformation in online discourse about critical issues, taking climate change as an example, by linking claims made on social media to relevant scholarly articles. Two datasets were constructed for the task [ AUR25 ] , the first consists of English climate-related claims gathered from existing resources, while the second is a corpus of 394,269 publications in climate and environmental sciences. The claims were filtered for scientific check-worthiness and rephrased to be atomic tweet-styled statements. A TREC-like [ Vo05 ] linking approach was then used to create claim-abstract pairs for annotation, resulting in a dataset of 435 unique claims, each connected to abstracts annotated as supports , refutes , or not enough information . 12 12 12 https://huggingface.co/datasets/rabuahmad/climatecheck Subtask I of ClimateCheck focused on retrieving the top 10 relevant abstracts per claim, evaluated using Recall@K as well as Binary Preference (Bpref), while subtask II asked participants to classify the relation of each claim-abstract pair they retrieved, evaluated using F1-score with additional scaling based on successful retrieval. The task was hosted on CodaBench [ Xu22 ] , attracting 27 participants with ten leaderboard submissions. The top team achieved a Recall@10 score of 0.66, Bpref of 0.49 (subtask I), and and F1 of 0.73 (subtask II). 2.4 FoRC2025: Field of Research Classification Organiser: Deutsches Forschungszentrum für Künstliche Intelligenz (DFKI) The second FoRC shared task [ Fr25 ] was hosted at

Read the complete text