Analyzing 16,193 LLM Papers for Fun and Profits
Zhiqiu Xia ∗ , Lang Zhu ∗ , Bingzhe Li † , Feng Chen † , Qiannan Li ‡ , Chunhua Liao § , Feiyi Wang ¶ , and Hang Liu ∗ · 2025
Read Analyzing 16,193 LLM Papers for Fun and Profits by Zhiqiu Xia ∗ , Lang Zhu ∗ , Bingzhe Li † , Feng Chen † , Qiannan Li ‡ , Chunhua Liao § , Feiyi Wang ¶ , and Hang Liu ∗ (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-08619.txt.
Introduction
Zhiqiu Xia and co-authors analyse more than sixteen thousand papers on large language models, in a 2025 paper. The flood of such papers is reshaping computer science, shifting what top conferences publish. The study covers seventy-seven leading conferences over six years, from 2019 to 2024, and looks at the trend from four angles.
First, it tracks how language model work drives topic shifts inside major conferences. Second, topic modelling maps which sub-areas are growing and what each conference cares about. Third, it compares the contribution patterns of universities and companies. Fourth, it studies how the country of origin shapes development paths. Pulling the strands together, the authors draw ten key insights into how the research ecosystem around these models is evolving. The paper is a map of a fast-moving field, drawn from its own publication record.
Catalog record
| CREATOR | Zhiqiu Xia ∗ , Lang Zhu ∗ , Bingzhe Li † , Feng Chen † , Qiannan Li ‡ , Chunhua Liao § , Feiyi Wang ¶ , and Hang Liu ∗ |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-08619
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Analyzing 16,193 LLM Papers for Fun and Profits Zhiqiu Xia ∗ , Lang Zhu ∗ , Bingzhe Li † , Feng Chen † , Qiannan Li ‡ , Chunhua Liao § , Feiyi Wang ¶ , and Hang Liu ∗ Abstract Large Language Models (LLMs) are reshaping the landscape of computer science research, driving significant shifts in research priorities across diverse conferences and fields. This study provides a comprehensive analysis of the publication trend of LLM-related papers in 77 top-tier computer science conferences over the past six years (2019–2024). We approach this analysis from four distinct perspectives: (1) We investigate how LLM research is driving topic shifts within major conferences. (2) We adopt a topic modeling approach to identify various areas of LLM-related topic growth and reveal the topics of concern at different conferences. (3) We explore distinct contribution patterns of academic and industrial institutions. (4) We study the influence of national origins on LLM development trajectories. Synthesizing the findings from these diverse analytical angles, we derive ten key insights that illuminate the dynamics and evolution of the LLM research ecosystem. I Introduction The recent rapid evolution of LLMs has significantly changed the research trend, especially in the domain of natural language processing (NLP). Over the past few years, LLMs such as BERT [ 1 ] , GPT-3 [ 2 ] , Llama [ 3 ] and their derivatives have demonstrated unprecedented capabilities in generating, understanding, and adapting to human language. These advancements have impacted not only core NLP domains but also extended their influence across a wide array of interdisciplinary applications, including computer vision, robotics, systems, and so on. LLMs have shown a paradigm shift, altering the research trajectories of various fields and conferences by applying LLMs. The growing availability of LLMs with increasing computational resources, datasets, and evaluations has spurred an exponential increase trend in the research community. This trend can be reflected in the increasing number of LLM-related publications from all kinds of leading computer science conferences. I-A Related Work & Motivations Consequently, bibliometrics—the quantitative examination of publication patterns, research trends, and topical emphases—offers valuable insights into the evolution of scientific disciplines. Numerous studies have analyzed publication trends, collaboration networks, and thematic shifts across various research domains. For instance, [ 4 ] analyzed 89 million scholarly publications from 1900 to 2015, revealing a notable transition towards collaborative research efforts over time. Within the NLP community specifically, recent studies have examined citation patterns and paradigm shifts in research topics. [ 5 ] analyzed citation trends within NLP literature, while [ 6 ] developed a causal discovery and inference-based framework to systematically trace the evolution and detect paradigm shifts in NLP research. More recently, the rise of LLMs has become a focal point in bibliometric analyses. For example, [ 7 ] conducted a comprehensive study covering over 5,000 publications related to LLM research from 2017 to early 2023, providing detailed insights into the evolution of foundational models, interdisciplinary application trends, and collaborative patterns. Similarly, [ 8 ] analyzed 16,979 arXiv publications on LLMs, highlighting shifts in topics, authorship patterns, and institutional collaborations. Their findings pointed to an increasing disciplinary shift toward societal implications of LLMs and a growing influx of researchers from outside traditional NLP fields. However, the scope of these previous bibliometric studies is inherently limited by their reliance on specific datasets, such as the ACL Anthology corpus or the arXiv repository. While these datasets provide valuable and specialized coverage, they tend to limit the bibliometric analysis to a particular field or preprint platforms, potentially missing broader disciplinary insights and introducing biases toward certain topics or trends. I-B Contributions This study dismantles these limitations by providing a comprehensive analysis of LLM-related research trends within virtually all major top-tier computer science (CS) conferences. By examining more than 160k+ published papers in 77 conferences across diverse subfields over the past six years, we identify 16,193 LLM-related papers capturing ten valuable insights. This paper’s contributions are as follows: 1. LLM-related papers’ publication trend of top conferences: We examine the LLM research landscape across leading CS conferences. This includes conferences spanning artificial intelligence (AI), systems, theory, and interdisciplinary areas that previous works have not comprehensively addressed. 2. Topics shifting driven by LLMs: We study topics shifting by topic modeling to provide comprehensive observations of topics in these subfields. This result echoes the recent societal concerns and hot topics in the research community. 3. Impact of academic and industrial contributions on LLM research: We explore how the academic and industry institutions shape the LLM research in these conferences. By analyzing publication trends, we study which institutions have the highest output and impact in the field of LLMs. 4. The role of nations in shaping research trends: We analyze how nations contribute to research development in different fields. This includes examining national influences on topic distribution. The remainder of this paper is organized as follows: Section II details the datasets and analytical methods employed in this study. Subsequently, we present and analyze our results in depth in Section III , building upon these findings to culminate in ten key insights that characterize the evolving landscape of LLM research. Finally, Section IV concludes the paper by summarizing our work and its implications. II Dataset and Methods II-A Identifying LLM-related Papers Our dataset contains 168,331 papers from 2019 to 2024 of 77 prestigious computer science and engineering conferences according to CSRankings [ 9 ] . These 77 conferences encompass fields from algorithms to emerging areas of computer science and engineering, covering four main avenues: AI, systems, theory, and interdisciplinary areas. Fig. 1: Number of LLM-related papers in each area. To identify LLM-related papers, [ 8 ] composes a set of LLM-related keywords and runs keyword matching throughout the title and abstract across all the papers in the dataset to pick LLM-related papers from them. However, we find an LLM-related paper might not have those keywords in its title or abstract. Further, it is challenging to pre-define a comprehensive set of keywords for LLM, which is a rapidly evolving field. Instead, we leverage the capabilities of LLMs themselves to identify LLM-related papers. Specifically, we define the criteria for what constitutes an LLM-related paper in the prompt. Subsequently, we deploy a local Llama3.1-8B-Instruct [ 10 ] model to determine whether a given paper is related to LLMs or not based on its title and abstract. Our selection process yields 16,193 LLM-related papers, with the numbers of papers in each area shown in Fig. 1 . Table I further shows the # of papers published in each year during 2019 - 2024. In 2019, there are only 503 LLM-related papers, indicating that research in this area was still in its early stages. And the number doubles in 2020 and climbs further in 2021. Beginning in 2022, there was a sharp increase in the number of LLM papers. The most significant surge occurred in 2024, with 7,109 LLM-related papers, representing an increase of 3,255 papers compared to 2023. This dramatic growth underscores how rapidly LLM is taking a central role in advancing CS research. TABLE I: Number of LLM-related papers from 2019 to 2024 Year 2019 2020 2021 2022 2023 2024 Number 503 1105 1595 2027 3854 7109 II-B Topic