CiteBART: Learning to Generate Citations for Local Citation Recommendation

Ege Yiğit ÇELİK, Selma TEKİR · 2024

Read CiteBART: Learning to Generate Citations for Local Citation Recommendation by Ege Yiğit ÇELİK, Selma TEKİR (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17534.txt.

Introduction

Ege Yiğit Çelik and Selma Tekir teach a model to suggest citations in a 2024 paper. When a draft has a gap where a source should sit, local citation tools propose papers that fit the spot. Older tools fetch and rerank lists, while fresh generative tools write the citation out.

Their model, CiteBART, pretrains by hiding author date tokens in text and learning to restore them. A global form also sees the citing paper title and summary, which sharpens the signal. That form topped most test sets and carried well to new sets. Fake citations stayed rare in top picks and fell further when a true pick was present.

Catalog record

CREATOREge Yiğit ÇELİK, Selma TEKİR
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE62 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17534

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CiteBART: Learning to Generate Citations for Local Citation Recommendation Ege Yiğit ÇELİK, Selma TEKİR Abstract Local citation recommendation (LCR) suggests a set of papers for a citation placeholder within a given context. The task has evolved as generative approaches have become more promising than the traditional pre-fetch and re-rank-based state-of-the-art approaches. This paper introduces citation-specific pre-training within an encoder-decoder architecture, where author-date citation tokens are masked to learn to reconstruct them to fulfill LCR. There are two variants for this pre-training. In the local context-only base scheme (CiteBART-Base), the citation token in a local context is masked to learn to predict the citation. The global version (CiteBART-Global) extends the local context with the citing paper’s title and abstract to enrich the learning signal. CiteBART-Global achieves state-of-the-art performance on LCR benchmarks except for the FullTextPeerRead dataset, which is quite small to see the advantage of generative pre-training. The effect is significant in the larger benchmarks, e.g., Refseer and ArXiv., with the Refseer benchmark-trained model emerging as the best-performing model. We perform comprehensive experiments, including an ablation study, a qualitative analysis, and a taxonomy of hallucinations with detailed statistics. Our analyses confirm that CiteBART-Global has a cross-dataset generalization capability; the macro hallucination rate (MaHR) at the top-3 predictions is 4%, and when the ground-truth is in the top-k prediction list, the hallucination tendency in the other predictions drops significantly. We publicly share our code 1 1 1 https://github.com/eyclk/CitationRecommendation , base datasets 2 2 2 https://drive.google.com/drive/folders/1WlqlTkSj8LwihbrQvBX5F9_0uZAGGhiE?usp=drive_link , global datasets 3 3 3 https://drive.google.com/drive/folders/1JH34nEXt8_p-0P9A--aQHK4yBXQfJe4v?usp=drive_link , and pre-trained models 4 4 4 https://drive.google.com/drive/u/2/folders/1OBg6W3kQw4VWPMfrXEPxN8LzTopR1jak to support reproducibility. K eywords Citation masking \cdot BART pre-training \cdot Local citation recommendation 1 Introduction Citations are essential building blocks in scientific writing. Their accurate placements indicate quality, as one should know the literature to claim contributions and put the current study in the context of the existing work from different aspects, such as background information, method, and result comparison ( Cohan et al., 2019 ) . Citation prediction is defined as a two-step process where the former focuses on where in the sentence to place the citation ( Buscaldi et al., 2024 ) , while the latter (citation recommendation) obtains a set of candidate papers once there is a specified citation placeholder in a given context. In this sense, citation recommendation serves as a citation suggestion mechanism. For a given scientific text, it can suggest additional papers on a similar topic. These suggestions can be considered additional reading material alongside the targeted paper, corresponding to the ground-truth citation. There are two levels of citation recommendation: the first, whom to cite, and the second, whom to cite in what context. The former is global citation recommendation, traditionally performed based on paper metadata such as author names, paper titles, abstracts, conference venues, publisher information, etc. Recently, custom citation-aware language models (SciBERT ( Beltagy et al., 2019 ) , SPECTER ( Cohan et al., 2020 ) ) learn good citation-aware embeddings for full papers to perform well in this task. The latter task is local citation recommendation (LCR), aiming to determine the target paper for a citation placeholder. Figure 1: CiteBART workflow. The yellow and green examples represent the workings of CiteBART-Base and CiteBART-Global, respectively. During inference, the expected outputs are in the author–date citation format, unlike the pre-training stage. LCR has been addressed in a few works. BERT-GCN ( Jeong et al., 2020 ) utilizes a feedforward neural network to combine local citation context representations using BERT with citation encodings through Graph Convolutional Neural Networks (GCN). The most recent solutions to the problem adopt a two-step process that consists of pre-fetching and re-ranking. DualEnh ( Medić and Snajder, 2020 ) enhances a local citation context with the citing article’s title and abstract and uses this enhanced context as the query vector to retrieve the most similar candidate articles using their titles and abstracts. It performs this ranking through BiLSTM representations of inputs with attention layers on top. On the other hand, HAtten ( Gu et al., 2022 ) initially pre-fetches a set of papers using the nearest neighbor search between local citation context extended with the citing paper’s title and abstract (query text as a whole) and the title and abstracts from a given pool of papers. Afterward, it re-ranks the selected candidate papers using a fine-tuned SciBERT ( Beltagy et al., 2019 ) model where the input is the query text concatenated with a candidate paper’s title and abstract. SymTax ( Goyal et al., 2024 ) improves upon HAtten by introducing an additional Enricher module and reranking candidate papers using taxonomical relationships along with contexts. The existing LCR works are not built upon Transformers but benefit from it indirectly, such as re-ranking the results (using fine-tuned SciBERT). Distinctively, we propose CiteBART, a custom pre-training approach based on the Transformer architecture. We mask citation tokens in the local contexts to learn to reconstruct them effectively during pre-training. Fierro et al. (2024) support information-seeking using query-focused summarization, responding to user queries by answers with source attributions. Attributions are in the form of in-line references to the passages. In a similar direction, the ALCE benchmark ( Gao et al., 2023 ) collects a diverse set of questions and retrieved passages to support answer generation with appropriate citations. As these models exhibit citation generation abilities, they are similar to CiteBART. However, CiteBART aims to fill in a passage with citations instead of targeting retrieval-based summaries with citations. In ALCE, there is a closed-book configuration where the model does not access any retrieved document to generate answers to a user query but is still different as the focus is answering a query instead of filling in the citation placeholder. Table 1: An example for input and target formats for evaluation with CiteBART. Due to space constraints, we present the contexts and abstracts in an abbreviated form. Strategy Input Target Base … error rate of 5.8% and a word error rate of 28.7%, which are on par with previous reported results <mask> . Unlike prior work, we do not use a language model during decoding and … Yao and Zweig, 2015 Global … error rate of 5.8% and a word error rate of 28.7%, which are on par with previous reported results <mask> . Unlike prior work, we do not use a language model during decoding and … </s> Deep Voice: Real-time Neural Text-to-Speech </s> We present Deep Voice, a production-quality text-to-speech system constructed entirely from deep neural … Yao and Zweig, 2015 In our approach, the base scheme (CiteBART-Base) learns through the masked citation context. In a second technique (CiteBART-Global), we extend the masked context with the citing paper’s global information, e.g., title and abstract (Table 1 ). Inspiring from pre-training under the REALM framework ( Guu et al., 2020 ) , we append this global information to the local context, allowing backpropagation through the global information to learn associations with the pool of papers from the corpus. CiteBART achieves superior performance without relying on a pre-fetch and re-rank pipeline. It is an end-to-end learning system. On the other hand, a pre-fetch and re-rank pipeline, such as HAtten, utilizes the citing papers’ titles and abstracts with the local contexts to form the query encoding and the

Read the complete text