In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis

Hiba Arnaout, Noy Sternlicht, Tom Hope, Iryna Gurevych · 2025

Read In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis by Hiba Arnaout, Noy Sternlicht, Tom Hope, Iryna Gurevych (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-14838.txt.

Introduction

Hiba Arnaout, Noy Sternlicht, Tom Hope, and Iryna Gurevych summarize research impact over time in two thousand twenty five. Citation counts miss how a paper truly shapes its field. Praise and critique leave different marks across the years. They propose impact summaries that track both kinds of citation.

Fine grained citation intents feed the summaries as they evolve. Confirmation and correction citations are told apart in the text. A tailored evaluation frame scores the summary quality. Human ratings correlate well on insight based measures. Professors who reviewed the output wanted more of it.

Catalog record

CREATORHiba Arnaout, Noy Sternlicht, Tom Hope, Iryna Gurevych
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE89 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-14838

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

In-depth Research Impact Summarization through Fine-Grained Temporal Citation Analysis Hiba Arnaout, Noy Sternlicht, Tom Hope, Iryna Gurevych Abstract Understanding the impact of scientific publications is crucial for identifying breakthroughs and guiding future research. Traditional metrics based on citation counts often miss the nuanced ways a paper contributes to its field. In this work, we propose a new task: generating nuanced, expressive, and time-aware impact summaries that capture both praise (confirmation citations) and critique (correction citations) through the evolution of fine-grained citation intents. We introduce an evaluation framework tailored to this task, showing moderate to strong human correlation on subjective metrics such as insightfulness. Expert feedback from professors reveals a strong interest in these summaries and suggests future improvements. Data and code are made available. 1 1 1 https://ukplab.github.io/arxiv2025-generating-impact-summaries 1 Introduction Figure 1: We propose a new task to summarize a paper’s evolving impact over time, by analyzing impact-revealing citation contexts, reflecting both praise (confirming ideas) and critique (calls for correction). Our summary of Koller and Sahami 1997 reveals its impact trajectory—adaptation, critique, and rediscovery—offering deeper insight than its \sim 1.4k cite count. Citation counts are a common proxy for measuring the impact of research papers, but they offer only a shallow view that fails to capture how a paper has influenced subsequent work. A raw citation count does not reveal whether a paper was foundational, extended, critiqued, or merely mentioned in passing. To truly understand the impact of a research paper, we must go beyond simple counts and examine the context in which it is cited, analyzing how its ideas have been discussed, applied, and evolved over time. Manually tracking how a paper is discussed across a very large number of publications and diverse domains is infeasible due to the scale and complexity of the task. To address this, we introduce a new task: research impact summary generation , which aims to generate concise, time-aware narratives that reflect a paper’s evolving scientific influence. These summaries can support practical use cases such as enhancing future research directions by helping researchers quickly assess a paper’s relevance and legacy, or assisting funding agencies, hiring committees, and research administrators in making more informed evaluations of scholarly contributions. Figure 1 shows an example of an impact summary about a research paper Koller and Sahami 1997 , published at The International Conference on Machine Learning in 1997. This paper’s impact has followed a dynamic trajectory, initially used for its method (1997–2007), later critiqued for scalability and accuracy limitations, then refined (2008–2018), and recently rediscovered as an inspiration for modern methods (2019–present). This example shows the need to look beyond the paper’s \sim 1.4k citations to understand the deeper story of how ideas gain relevance, face scrutiny, and find renewed significance. We break the task of generating impact summaries for research papers (Section 2.1 ) into two subtasks. First, given all the citation contexts of a paper, we identify the “impact revealing” citations that directly interact with the paper, and their specific intents, whether in confirmation of its ideas (e.g., “method use”) or in correction (e.g., “identifying limitations and proposing refinements”). Second, using only the impact-revealing citation contexts, their dates and intents, we generate an impact summary that captures the paper’s influence over time. To the best of our knowledge, this is the first attempt to express scientific impact through time-aware textual summaries derived from citation context analysis. An overview is shown in Figure 1 . To implement these subtasks, our method, in Section 2 , uses in-context learning (ICL) with Large Language Models (LLMs) to generate fine-grained citation intents in the first stage. In the second stage, it filters for impact-revealing citation contexts based on the generated intents and feeds them to an LLM to generate semi-structured impact summaries under a pre-defined schema. Existing work on scientific impact analysis predominantly focuses on citation and similar numerical counts as the main indicator of impact Gu and Krenn 2024 , ignoring the exact reasons a paper has been cited, which can often be “unimpactful”, e.g., citing a paper to acknowledge background work on a topic. Work on citation context analysis Lauscher et al. 2022 studies the role citations play, but is based on coarse categories and is not directly intended for impact analysis. The goal of our new task is to create a narrative about a paper’s impact using fine-grained citation intent analysis. Moreover, existing work typically emphasizes praise (confirmatory citations) Zhang et al. 2024 ; Valenzuela et al. 2015 , but scientific progress relies on both confirmation and correction Catalini et al. 2015 , as ideas advance through adoption as well as critique and refinement. In this work, we ensure that our generated impact summaries capture both aspects of impact (when applicable), as shown in Figure 1 . To support this broader view of impact, we extend the existing PST-Bench data set Zhang et al. 2024 , which only focuses on “motivation” and “inspiration” intentions to consider an impact revealing citation, by introducing, in Section 2.5 , a new ground-truth data set of 4000 citation contexts labeled for whether they reveal impact. Our extended dataset also covers not only additional intents of confirmatory citations (e.g., method use), but also correction citations (e.g., method refinement). For detailed comparisons with related work, see Appendix A and Section 5 . Finally, in the absence of gold-standard summaries for this novel impact generation task, we propose an automated evaluation framework to measure the trustworthiness and informativeness of our impact summaries (Section 2.4 ) and demonstrate a moderate to strong correlation with human assessments (Section 3.3 ). To further validate our approach, we collect expert feedback from professors on the usefulness of impact summaries about their own papers (Section 4 ), demonstrating the need for such summaries. For future research, we release the code and data used to develop this work. 2 Method 2.1 Task definition We propose the task of generating time-aware impact summaries for research papers by identifying and analyzing impact-revealing citation contexts and their intents over time. Definition 1 . Citation context refers to the specific part of a paper p^{\prime} where another paper p is mentioned, including the surrounding text that explains how p is relevant to p^{\prime} . Definition 2 . Citation intent refers to the purpose (the “why”) behind citing paper p in paper p^{\prime} , normally inferred from the citation context. Definition 3 . An Impact-revealing citation intent is a citation intent where the cited paper p has a direct influence on the citing paper p^{\prime} through: (i) Confirmation , by adopting or building upon p ; or (ii) Critique , by correcting or refining p . Citation intents that do not fall under these categories are considered non-impact-revealing. Figure 1 shows the intent “acknowledge background work”, for the context that simply refers to prior work in a general manner, without actively building upon or challenging it. Hence, this is not considered impact-revealing. In contrast, the intent “method used”, for the context describing the implementation of a popular approach is indeed impact-revealing. Definition 4 . A scientific impact summary of a research paper describes the impact paper p has had on subsequent research. In other words, how it has been directly used for both confirmation

Read the complete text