Freshness and Informativity Weighted Cognitive Extent and Its Correlation with Cumulative Citation Count

Zihe Wang, Jian Wu · 2024

Read Freshness and Informativity Weighted Cognitive Extent and Its Correlation with Cumulative Citation Count by Zihe Wang, Jian Wu (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-03557.txt.

Introduction

Zihe Wang and Jian Wu refine a measure of research breadth in a 2024 paper. Cognitive extent counts how many distinct phrases appear in a body of writing, a proxy for how wide its ideas range. Their new version weights each scientific term by two factors: how fresh it is, and how much information it carries.

A term's life is modeled from how often it appears in print over time, fit by a blend of bell curves. Freshness compares its use at print time to its whole life total. Information weight scales by how rare the term is across titles. On a large language research archive, the new score tracked later citation counts.

Catalog record

CREATORZihe Wang, Jian Wu
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE22 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-03557

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Freshness and Informativity Weighted Cognitive Extent and Its Correlation with Cumulative Citation Count Zihe Wang, Jian Wu Abstract In this paper, we revisit cognitive extent, originally defined as the number of unique phrases in a quota. We introduce Freshness and Informative Weighted Cognitive Extent (FICE), calculated based on two novel weighting factors, the lifetime ratio and informativity of scientific entities. We model the lifetime of each scientific entity as the time-dependent document frequency, which is fit by the composition of multiple Gaussian profiles. The lifetime ratio is then calculated as the cumulative document frequency at the publication time t 0 t_{0} divided by the cumulative document frequency over its entire lifetime. The informativity is calculated by normalizing the document frequency across all scientific entities recognized in a title. Using the ACL Anthology, we verified the trend formerly observed in several other domains that the number of unique scientific entities per quota increased gradually at a slower rate. We found that FICE exhibits a strong correlation with the average cumulative citation count within a quota. Our code is available at https://github.com/ZiheHerzWang/Freshness-and-Informativity-Weighted-Cognitive-Extent [orcid=0009-0007-4293-3449, [email protected], ] 1 [orcid=0000-0003-0173-4463, [email protected], url=https://www.cs.odu.edu/ jwu/, ] *1 1 keywords cognitive extent, citation impact, entity recognition, document frequency † † copyrightyear: 2024 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: IMSC Workshop@JCDL 2024, December 20, 2024, Hong Kong, CN † † address: Ohio State University, Columbus, OH, United States † † address: Old Dominion University, Norfolk, VA, United States † † corresp: Corresponding author. † † These authors contributed equally. 1 Introduction Cognitive extent is an approach to quantify the extent of cognitive domains of scientific fields based on the concept of lexical diversity Milojević 2015 . The metric was originally calculated by counting the number of unique concepts (phrases) appearing in the titles of statistically large unit quotas of scientific literature, which reflects the extent of the cognitive territory covered in that literature. Cognitive extent has been used as a representation of knowledge gained by scientists. It has been shown that cognitive extent calculated in multiple academic fields (Physics, Astronomy, and Biomedicine) grew at a slower rate. However, this definition of cognitive extent has two limitations. First, it only accounts for the occurrence of a phrase as a dichotomous value within a quota and ignores when the phrase occurs. Specifically, a phrase is novel when it appears the first time in the title of Paper A, but when it appears again in the title of Paper B, it still maintains a level of freshness, and thus still reflects the scientist of Paper B’s cognitive knowledge except that the knowledge is no longer new. Second, the definition treats all phrases with the same weight. However, certain phrases may be more informative than others. For example, at a particular time the phrase “entity recognition” occurred in many titles but the phrase “nuclearity rhetorical relation” occurred in only a small number of titles. From a reader’s perspective, the latter phrase is more informative because the former phrase has been seen in many papers. To overcome the limitations, we propose Freshness and Informativity Weighted Cognitive Extend (FICE), which is calculated as a weighted occurrence of disambiguated unique scientific entities extracted from paper titles in a quota. Different from the original cognitive extent, FICE accounts for contributions of freshness and informativity of scientific entities extracted from documents within a corpus. The freshness is based on the lifetime ratio and the informativity is based on time-dependent document frequency across scientific entities in a document. Here the document can be any form of scholarly text. Throughout this paper, we focus our study on paper titles. We verified a previous finding using ACL Anthology papers that although the number of papers increased exponentially, the unique number of scientific entities per quota gradually increased with time at a slower rate. One property of FICE is its relationship with citation impact factors. We found that FICE exhibits a strong correlation with the logarithm of 5-year average cumulative citation counts for papers in the ACL Anthology. 2 Related Work The definition of cognitive extent is closely relevant to lexical diversity, defined as the extent of vocabulary disparity within a given language sample Bérubé et al. 2018 . Early metrics determined the lexical diversity only by single words Jarvis and Daller 2013 ; Malvern et al. 2004 . Berube et al. proposed the type-token ratio as a metric of lexical diversity Bérubé et al. 2018 . Both methods were based on word-level tokens and ignored their connections, therefore they do not necessarily reflect the knowledge , which is better captured by phrases and entities. Milojević was the first to propose using concepts (phrases) to quantify the extent of cognitive domains of scientific fields Milojević 2015 . This method was later applied to study the properties of paper titles in various domains Milojević 2017 . Recently, a method was proposed to use neural embedding of paper titles to represent the cognitive content of a cluster of papers Eykens et al. 2023 . Although neural embedding has been widely used to capture the semantics of text and compare the semantic similarities between two pieces of text, the embedding itself cannot be directly converted to a numeral that represents the cognitive extent of a single or a corpus of documents. Bibliographic impact factors have been extensively studied and several citation impact factors have been proposed Waltman 2016 . The total number of citations (raw citations) is usually criticized as a good indicator because of domain discrepancies, data completeness, and other random factors. The average number of citations per publication of a research unit is frequently used but also criticized because the average value can be biased due to the skewness of citation distribution within the research unit Albarrán et al. 2011 . However, it was argued that short-term citations can be considered as currency on the research front and long-term citations can contribute to the codification of knowledge claims into bodies of knowledge Leydesdorff et al. 2016 . Determining the exact boundary between short-term and long-term is non-trivial. Here we adopt C_{5}(y) , which is the 5-year average cumulative citation count of papers within a quota as an estimate of the quota’s average impact over the short- and long-term. 3 Methodology 3.1 Scientific Entity Recognition and Disambiguation A scientific entity is defined as a noun phrase that delivers domain knowledge of interest Wu et al. 2020 ; Hong et al. 2020 . Scientific entities can be extracted using sequence tagging models, and constructing knowledge graphs, e.g., Li et al. 2020 ; Hong et al. 2020 . Recently, large language models (LLMs) have shown superior performance on named entity recognition, e.g., Wang et al. 2023 ; Sousa et al. 2023 . Two scientific entities may have similar semantics. To investigate the impact of entity disambiguation on the results, we conflate entities with similar semantic meanings. This was treated as a classification problem using a thresholded method on the similarity scores calculated using the Cross Encoder Li et al. 2020 , a model that takes two entity names and outputs a similarity score. The threshold was calibrated

Read the complete text