Citation Parsing and Analysis with Language Models
parth sarin, Juan Pablo Alperin · 2025
Read Citation Parsing and Analysis with Language Models by parth sarin, Juan Pablo Alperin (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15948.txt.
Introduction
Parth Sarin and Juan Pablo Alperin tested open-weight language models in twenty twenty-five as citation parsers, aiming at global equity in scholarly indexing. Tracking how knowledge circulates requires machine-readable citations, and research from the Global South is under-indexed partly for lack of such tools — an exclusion that reinforces itself. The pair assembled a data set of citations in plain text matched with their annotated components.
Even without fine-tuning, current language models identified citation fields with high accuracy, beating specialized state-of-the-art parsers. The smallest model tested, well under a billion parameters, parsed all fields accurately within thirty-two passes. The authors argue cheap post-training could yield small, robust parsers — and with them, fairer citation networks, better discovery, and stronger metascience worldwide.
Catalog record
| CREATOR | parth sarin, Juan Pablo Alperin |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-15948
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Citation Parsing and Analysis with Language Models parth sarin, Juan Pablo Alperin Abstract A key type of resource needed to address global inequalities in knowledge production and dissemination is a tool that can support journals in understanding how knowledge circulates. The absence of such a tool has resulted in comparatively less information about networks of knowledge sharing in the Global South. In turn, this gap authorizes the exclusion of researchers and scholars from the South in indexing services, reinforcing colonial arrangements that de-center and minoritize those scholars. In order to support citation network tracking on a global scale, we investigate the capacity of open-weight language models to mark up manuscript citations in an indexable format. We assembled a dataset of matched plaintext and annotated citations from preprints and published research papers. Then, we evaluated a number of open-weight language models on the annotation task. We find that, even out of the box, today’s language models achieve high levels of accuracy on identifying the constituent components of each citation, outperforming state-of-the-art methods. Moreover, the smallest model we evaluated, Qwen3-0.6B, can parse all fields with high accuracy in 2 5 2^{5} passes, suggesting that post-training is likely to be effective in producing small, robust citation parsing models. Such a tool could greatly improve the fidelity of citation networks and thus meaningfully improve research indexing and discovery, as well as further metascientific research. 1 Introduction Science is not an equal playing field. There is a large—though shrinking—inequality between knowledge production in the Global North and Global South. To properly understand this gap, national research capacities, and the broader power dynamics in global research contributions, it is necessary to have high quality and complete indexes of scholarly production. Accurate and complete metascientific information is also central to decision making at every level of the research system from the tenure and promotion decisions about individual faculty, to institutional resource allocation, and to national science policies. Such information is not indexed today: It has been observed for decades that research in the Global South is significantly underrepresented in supposedly global indexes, databases, and search engines for scholarly work ( Cetto et al. 1998 ; Khanna et al. 2022 ; Mongeon & Paul-Hus 2016 ) . So, for over 50 years, decisions about funding, tenure, collaboration, and governance have been made without the same type of reliable quantitative data that exists to tabulate research in the Global North. Fortunately, a number of projects are seeking to address global gaps in indexing and discoverability. New bibliographic databases like OpenAlex are taking a more inclusive indexing approach that has allowed them to significantly outperform existing databases in terms of coverage ( Alperin et al. 2024 ; Culbert et al. 2024 ; Jiao et al. 2023 ) . However, despite the significantly greater coverage of works, including those from the Global South, there continues to be an enormous gap in the indexing of references ( Alperin et al. 2024 ; Culbert et al. 2024 ) . While more complete bibliographic records are useful for understanding knowledge production, references and citations allow us to better understand the connections and circulation of knowledge, and are thus an essential aspect of research information. This is the project to which we seek to contribute. Namely, there has been recent interest in developing new citation parsing techniques that improve on state-of-the-art methods like GROBID and Crossref search to be performant in more languages and trainable end-to-end ( Choi et al. 2023 ; Joshi et al. 2023 ) . Though they were created through very extractive and neocolonial processes ( Gray & Suri 2019 ; Hao 2024 ) , modern language models can indeed operate in multiple languages and modalities and they are end-to-end trainable. Some smaller language models can also run in low-compute environments, and even on mobile devices or in the browser. In this paper, we evaluate open-source, decoder-based language models on the task of citation parsing. Specifically, we aim to address the following research questions: 1. How accurate are open-weight language models at identifying different components of a citation? How do they compare to state-of-the-art methods for reference parsing? 2. Are small language models — which can run in the browser or on a device — promising for this task? 2 Methods 2.1 Dataset We began by assembling a dataset of citations in plain, formatted text, along with the same citations marked up in JATS format. We drew from two existing corpora. First, Garnett 2016 assembled the “XML Markup Evaluation Corpus,” for the Public Knowledge Project (PKP) which includes 829 submitted article manuscripts marked up in JATS format. Second, Open Research Europe (ORE) is a publishing platform with an annotated corpus of 848 articles, marked up in JATS format ( European Commission 2025 ) . We extracted the marked up version of each citation from the reference list in the JATS XML markup and matched it to a (markdown-formatted) plaintext citation extracted from the article. Specifically, we converted each article to markdown using the markitdown tool ( Microsoft Corporation 2025 ) . Then, from that markdown, we extracted plaintext citations using Llama-3.1-8B-Instruct prompted with the system prompt in Appendix A.1 . We programmatically verified that each of the citations appeared in the article to prevent hallucinated citations. (a) Garnett/PKP Corpus (b) ORE Corpus Figure 1: Similarity distributions for Garnett/PKP and ORE corpora Then we matched the plaintext citations (the output of the language model) with the marked-up citations (provided in the XML files in each of the corpora). Let p be the plaintext citation, c be a text-only version of the marked-up citation, and d(p,c) be the edit distance between the two. Take |x| to represent the length (number of characters) of the string x . Define the similarity between p and c to be s(p,c)=1-\frac{d(p,c)}{\max\{|p|,|c|\}}. We matched citations in order of decreasing similarity and excluded all matches with a similarity score less than 0.75 . Figure 1 shows the distribution of similarity scores on the two corpora. Applying this procedure, we produced 13,149 matches from the Garnett/PKP corpus and 24,934 from the ORE corpus. We sampled 1,000 citations from each for a final dataset of 2,000 matched citations in plain text and marked up form. We also qualitatively analyzed the matching procedure, specifically to investigate why the average similarity score is lower for the ORE corpus than the Garnett/PKP corpus. Our suspicion is that this happened because the ORE corpus is for published articles and references in that corpus can contain additional links which don’t appear in the JATS markup. For example, here is one plaintext reference with a low similarity score: Viswanathan M, Ammerman A, Eng E, et al.: Community-based participatory research: assessing the evidence. Evid Rep Technol Assess (Summ). AHRQ Publication N° 4-E022-1. Agency for Healthcare Research and Quality, Rockville, MD, 2004; (99): 1–8. PubMed Abstract | Free Full Text In the published article, the highlighted section was linked to the PubMed abstract and full text of the article. The matched JATS annotation for this citation is correct but does not include those links, leading to a low score: <mixed-citation publication-type="journal"> <person-group person-group-type="author"> <name name-style="western"> <surname>Viswanathan</surname> <given-names>M</given-names> </name> <name name-style="western"> <surname>Ammerman</surname> <given-names>A</given-names> </name> <name name-style="western"> <surname>Eng</surname> <given-names>E</given-names> </name> <etal />