Evaluating authorship disambiguation quality through anomaly analysis on researchers’ career transition

Huaxia Zhou, Mengyi Sun · 2024

Read Evaluating authorship disambiguation quality through anomaly analysis on researchers’ career transition by Huaxia Zhou, Mengyi Sun (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-18757.txt.

Introduction

Huaxia Zhou and Mengyi Sun use a strange pattern in career data to grade how well databases tell same-named researchers apart. Their 2024 paper works with OpenAlex records for 5.8 million biomedical researchers. They estimated when each person first became an independent lead researcher, marked by appearing as last author.

Over sixty percent seemed to be lead authors in their very first career year, which is not plausible. The authors argue this glitch flags sloppy name matching. Researchers lacking affiliation data showed the glitch most, while those with ORCID identifiers showed it least. Before 2010, records for women were matched worse than those for men, which clouds older studies of gender gaps.

Catalog record

CREATORHuaxia Zhou, Mengyi Sun
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-18757

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Evaluating authorship disambiguation quality through anomaly analysis on researchers’ career transition Huaxia Zhou, Mengyi Sun Abstract Authorship disambiguation is crucial for advancing studies in science of science. However, assessing the quality of authorship disambiguation in large-scale databases remains challenging, since it is difficult to manually curate a gold-standard dataset that contains disambiguated authors. Through estimating the timing of when 5.8 million biomedical researchers became independent Principal Investigators (PIs) with authorship metadata extracted from the OpenAlex—the largest open-source bibliometric database—we unexpectedly discovered an anomaly: over 60% of researchers appeared as the last authors in their first career year. We hypothesized that this improbable finding results from poor name disambiguation, suggesting that such an anomaly may serve as an indicator of low-quality authorship disambiguation. Our findings indicated that authors who lack affiliation information, which makes it more difficult to disambiguate, were far more likely to exhibit this anomaly compared to those who included their affiliation information. In contrast, authors with Open Researcher and Contributor ID (ORCID)—expected to have higher quality disambiguation—showed significantly lower anomaly rates. We further applied this approach to examine the authorship disambiguation quality by gender over time, and we found that the quality of disambiguation for female authors was lower than that for male authors before 2010, suggesting that gender disparity findings based on pre-2010 data may require careful reexamination. Our results provide a framework for systematically evaluating authorship disambiguation quality in various contexts, facilitating future improvements in efforts to authorship disambiguation. Introduction Large-scale bibliometric databases form the backbone of quantitative studies in the science of science [ 1 ] . These comprehensive repositories—collecting scholarly works and associated metadata such as authorship, affiliations, and disciplinary classifications—enable investigations into research productivity [ 2 ] , collaboration networks [ 3 ] , and the impact of scholarly works [ 4 ] . However, the precision and reliability of such analyses critically depend on the quality of authorship disambiguation. Variations in spelling, cultural naming conventions, and incomplete metadata could cause multiple distinct individuals to be conflated under a single author profile or, conversely, a single individual to be split across multiple profiles [ 5 ] . Authorship disambiguation errors can bias findings about researcher careers, collaboration networks, and institutional or disciplinary dynamics. Although different algorithms and tools have emerged to address these challenges [ 6 , 7 , 8 ] , evaluating the quality of disambiguation at scale remains difficult. Manual curation of large and representative datasets as a gold standard is rarely feasible due to the challenge of unbiased sampling of researchers with diverse demographic backgrounds. For example, ORCID [ 9 ] , a self-reported authorship database that has been widely used as the gold standard for authorship disambiguation algorithms, suffers from highly uneven registration rate across countries [ 10 ] . In this study, we identify a novel, data-driven signature of poor authorship disambiguation quality. Our analysis centered on typical career progression in science, the transition from an apprenticeship to independent Principal Investigator (PI) status, often estimated from the first instance as the last author on a publication. We employed “last author analysis” approach [ 11 ] , collecting relevant metadata from the OpenAlex [ 12 ] —the largest open-source bibliometric database—to estimate when approximately 5.8 million biomedical researchers became independent PIs. Contrary to expectations, we found a striking irregularity: more than 60% of these researchers appeared as the last authors during their career debut years of publishing. Such a pattern is almost improbable under practical career trajectories, and its persistence across multiple years strongly suggests a systematic issue in authorship disambiguation. Further analyses confirmed that this anomaly was more common among authors lacking essential metadata—such as affiliation information or an ORCID—making them more challenging to reliably identify. Moreover, extending our examination to the dimension in gender revealed that data on female authors—particularly before 2010—were more frequently affected by these anomalies, raising concerns about the validity of historical analyses on gender disparities. To address this striking anomaly finding, our study provides a framework for diagnosing and quantifying the authorshup disambiguation quality in bibliometric databases. Through extending the reliance on manually curated datasets, our approach identifies systematic biases and provides actionable insights for improving metadata quality at author level. Given the widespread use of bibliometric analysis in shaping science policy, informing funding decisions, and guiding scholarly evaluation, ensuring the accuracy and reliability of foundational data is crucial. Our findings contribute to advance the robustness and fairness of large-scale empirical research in the science of science. Figure 1 : The flow of execution for estimate the time to career independence for researchers in biomedical fields shows a stiking anomaly. a. In the authorship conventions of academic publishing—particularly in biomedical research—the last author is often recognized as the Principal Investigator (PI) or senior researcher, indicating a level of career independence. The number of years it takes to achieve career independence can be estimated by calculating the difference between the career debut year—the year of a scholar’s first publication—and the career independence year—the year of their first publication as the last author. This approach provides a measurable estimation of their progression within the career trajectory. b. An unusual anomaly in the estimation of career independence within the biomedical field has persisted over multiple years. Specifically, over 62% of authors published their first last-author paper in the same year they debuted with their first publication. This unexpected pattern of immediate career independence remained consistent across cohorts that began in different years, suggesting a systemic issue rather than isolated incidents. Results The Abnormally high rate of obtaining career independence in the career debut year To perform “last author analysis”, we processed the snapshot of OpenAlex database to obtain the career trajectories of biomedical researchers who began their careers after 1999 (see Materials and Methods ). Specifically, we define a biomedical researcher as an author who publishes journal articles classified under either Biology or Medicine at Level 0 Concept within the OpenAlex taxonomy. Since in biomedical disciplines career independence is typically indicated by being the last author on a journal article, we then narrowed our focus on scholars publishing in these fields. We defined an individual’s career inception as the year of their earliest publication in the database and focused our analysis on authors with first publications after 1999, following the tradition of scientific career analysis [ 13 , 14 ] . This time frame mitigates the issue of missing authorship records stemming from PubMed’s former policy of capping the number of listed authors [ 15 ] —a limitation that affected OpenAlex, where PubMed is a primary source for biomedical literature [ 12 ] . In total, we obtained the career transition of 5,803,782 authors from their journal publication records. We retrieved 48,000,188 journal articles associated with this cohort of authors. The career transition to secure a tenure-track position in the biomedical research field is notably longer than in other disciplines, primarily due to the bottleneck created by the ’postdoc queue’ [ 16 ] . This phenomenon reflects the extended periods post-Ph.D. that researchers often spend in postdoctoral positions before obtaining tenure-track roles, navigating a competitive and saturated job market. Focusing on this cohort of authors who had published at least one paper as the last author,

Read the complete text