Presenting a classifier to detect research contributions in OpenAlex
Nick Haupka · 2025
Read Presenting a classifier to detect research contributions in OpenAlex by Nick Haupka (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-22479.txt.
Introduction
Nick Haupka presents a classifier that sorts research from non-research content in OpenAlex in a 2025 paper. OpenAlex is a major open catalog of scholarly works. Its article and review sets contain editorials, letters, and other paratexts mixed with true research, which can skew bibliometric studies.
Trained on open metadata, the classifier scores very high on standard accuracy measures. Applied to tens of millions of records, it reclassified about a tenth as non-research. That is a large data quality gain for a free resource. Scholars who measure science itself can clean their samples before drawing conclusions about fields or institutions.
Catalog record
| CREATOR | Nick Haupka |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-22479
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Presenting a classifier to detect research contributions in OpenAlex Nick Haupka Abstract Abstract This paper introduces a document type classifier with the purpose to optimise the distinction between research and non-research journal publications in OpenAlex. Based on open metadata, the classifier can detect non-research or editorial content within a set of classified articles and reviews (e.g. paratexts, abstracts, editorials, letters). The classifier achieves an F1-score of 0,95, indicating a potential improvement in the data quality of bibliometric research in OpenAlex when applying the classifier on real data. In total, 4.589.967 out of 42.701.863 articles and reviews could be reclassified as non-research contributions by the classifier, representing a share of 10,75%. Keywords : OpenAlex, Document types, machine learning, classification, bibliometrics 1 Introduction Bibliometric databases like OpenAlex ( Priem et al. (2022) ) , Crossref ( Hendricks et al. (2020) ) and Semantic Scholar ( Kinney et al. (2023) ) index millions of research publications each year. In this process, the document type plays a decisive role in distinguishing different types of works such as editorials, case reports, journal articles and reviews. The decisions made by database providers to classify publications into document types have a profound impact on numerous bibliometric procedures and performance evaluations. For example, university rankings like the Leiden Ranking Open Edition ( Van Eck & Waltman (2024) ) restrict their performance analysis on certain document types such as articles and reviews, while excluding document types like editorials and letters. Another measurement that relies on document types given by scholarly databases is the Journal Impact Factor. The Journal Impact Factor represents the influence of a journal by counting citations of articles and reviews published in a journal 1 1 1 https://support.clarivate.com/ScientificandAcademicResearch/s/article/Journal-Citation-Reports-Document-Types-Included-in-the-Impact-Factor-Calculation?language=en_US . Document types are also frequently used as a restriction criterion for research publications, for example when analysing the dissemination of open access in scholarly communication ( Piwowar et al. (2018) ) or when examining the influence of transformative agreements on hybrid journals ( Jahn (2025) ) . However, the typology to describe works in bibliometric databases as well as the methods used to assign document types vary between data sources and providers. A study by Haupka, Culbert et al. (2024) shows that OpenAlex tends to overestimate the assignment of the document type article when comparing it to Scopus, Web of Science, Semantic Scholar and PubMed. Alperin et al. (2024) made a similar observation, noting deviations in the classification of document types between OpenAlex and Scopus. Studies also show that the methods for applying document type classification by scholarly data providers are not always precise as the document type assigned within a database can sometimes differ from the actual document type. For instance, Donner (2017) investigated a sample of document types retrieved from Scopus and Web of Science and compared them to the official publisher websites and found discrepancies in more than 17% of the cases. Sometimes, document types are also not covered in bibliographic databases. For example, an analysis of data journals by Jiao et al. (2023) has shown that OpenAlex lacks the classification of data papers which are assigned to an independent document type in Scopus and Web of Science but considered as journal articles in OpenAlex. Delgado-Quirós & Ortega (2024) compared typologies of document types of eight different bibliometric databases, including the databases OpenAlex, Crossref, The Lens, Scilit and Dimensions and found a high degree of consistency between document types retrieved from these databases, which can be attributed to Crossref as the main source of all databases mentioned. In the past year, the team behind OpenAlex, OurResearch, has announced and implemented several changes to its document type classification 2 2 2 https://github.com/ourresearch/openalex-guts/blob/main/files-for-datadumps/standard-format/RELEASE_NOTES.txt . First and foremost, it included the classification of preprints and reviews. Further it integrated document types from PubMed to relabel a set of articles as editorials and letters. Also, it extended their list-based approach for detecting paratexts and editorial material. The effects of this changes were documented by Haupka, Dörner & Jahn (2024) . While the implementation of document types from PubMed reduces the amount of works with the type article , a gap between the document type classification of OpenAlex and proprietary databases like Scopus and Web of Science still remains. In recent years, the application of machine learning algorithms have become increasingly popular in open science and especially in bibliometric databases ( Jeangirard (2022) ) . A prominent example is Semantic Scholar which uses machine learning classifiers for, among others, author disambiguation and field of study classification ( Kinney et al. (2023) ) . Another example is OpenAlex which uses a probability model to detect the language of a work. Further it uses machine learning classifier to match records with topics and SDGs 3 3 3 https://docs.openalex.org/api-entities/works/work-object . Also, the French Open Science Monitor uses machine learning algorithm to detect datasets and software ( Bracco et al. (2025) ) . In line with these developments, I have developed a document type classifier that filters research contributions in OpenAlex based on open metadata such as author count and number of references ( Haupka (2024) ) . Similar efforts were made by Charalampous & Knoth (2017) who created a classifier that takes metadata into account to distinguish between papers, theses and slides in (open access) repositories. Metadata for the classifier were extracted and aggregated from PDFs. They consisted of the number of authors, the total number of words in a document, the number of pages and the average number of words per page. Overall, this approach resulted in a high F1-score (0.96). A similar approach was conducted by Caragea et al. (2016) who extracted file and text specific features from documents to differentiate between books, slides, theses and papers using machine learning techniques. A more traditional approach to address the issue of misclassification of journal articles was proposed by Maisano et al. (2025) . It uses a concordance matrix to identify misclassified documents between two bibliographic databases (in this case Web of Science and Scopus). The methodology involves a manual control of document types which makes it difficult to apply on large amounts of journal publications. In this paper, I want to describe my classifier and my approach in depth. My results were previously briefly documented in a blog post in October 2024 4 4 4 https://subugoe.github.io/scholcomm_analytics/posts/oal_document_types_classifier/ . Since then, I implemented minor changes to the classifier. Therefore, I wanted to update my results. 2 Data Creating a gold standard dataset for validating document types is difficult. Data sources like Crossref, OpenAlex and PubMed mostly rely on publisher information when classifying document types 5 5 5 https://crossref.gitlab.io/knowledge_base/docs/topics/content-types/ 6 6 6 https://pubmed.ncbi.nlm.nih.gov/help/ . As Haupka, Culbert et al. (2024) has demonstrated, this can lead to errors in databases because each publisher can decide on his own how he classifies a publication. Also, data sources like Crossref and PubMed apply different taxonomies which makes it difficult to compare document types from different data sources. For this analysis, I decided to use document types from PubMed as my previous analysis has shown that the data source reaches comparable accuracy against Scopus and Web of Science in regard to the classification of research articles and reviews ( Haupka, Culbert et al. (2024) ) . Because, I