BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text
Ibrahim Al Azher †, Miftahul Jannat Mokarrama †, Zhishuai Guo † · 2025
Read BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text by Ibrahim Al Azher †, Miftahul Jannat Mokarrama †, Zhishuai Guo † (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18207.txt.
Introduction
Ibrahim Al Azher and co-authors built BAGELS, a test set for a quiet part of science writing: the limits of a study. Their 2025 paper notes that authors often soft-pedal weak points in their work. Clear limits help readers judge results and help others repeat the work. With paper counts rising fast, hand reading no longer scales.
The team pulled limit sections from papers in three fields and added notes from outside reviews. They then built methods that draft limits for a paper with the help of search-based prompts. A fine-grained scoring frame checks the drafts for fit and depth. The work also tests how well the scores themselves match human judgment. The full set gives later teams a shared base for tools that surface study limits on their own.
Catalog record
| CREATOR | Ibrahim Al Azher †, Miftahul Jannat Mokarrama †, Zhishuai Guo † |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 57 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18207
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
BAGELS: Benchmarking the Automated Generation and Extraction of Limitations from Scholarly Text Ibrahim Al Azher †, Miftahul Jannat Mokarrama †, Zhishuai Guo † Abstract In scientific research, “limitations” refer to the shortcomings, constraints, or weaknesses of a study. A transparent reporting of such limitations can enhance the quality and reproducibility of research and improve public trust in science. However, authors often underreport limitations in their papers and rely on hedging strategies to meet editorial requirements at the expense of readers’ clarity and confidence. This tendency, combined with the surge in scientific publications, has created a pressing need for automated approaches to extract and generate limitations from scholarly papers. To address this need, we present a full architecture for computational analysis of research limitations. Specifically, we (1) create a dataset of limitations from ACL, NeurIPS, and PeerJ papers by extracting them from the text and supplementing them with external reviews; (2) we propose methods to automatically generate limitations using a novel Retrieval Augmented Generation (RAG) technique; (3) we design a fine-grained evaluation framework for generated limitations, along with a meta-evaluation of these techniques. Code and datasets are available at: Code: https://github.com/IbrahimAlAzhar/BAGELS_Limitation_Gen Dataset: https://huggingface.co/datasets/IbrahimAlAzhar/limitation-generation-dataset-bagels † † Accepted to the Findings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025). 1 Introduction Figure 1: System architecture for dataset creation, limitation generation, and evaluation. In scientific articles, “limitations” refer to the inherent shortcomings, constraints, or weaknesses of a study that may influence its results or restrict the generalizability of its findings Ross and Bibler Zaidi (2019) . Such limitations can arise from various aspects of the research process, including the methodology, theoretical framework, data collection, experimentation, and analysis Ioannidis (2007) . Authors commonly acknowledge issues such as internal validity concerns, measurement errors, confounding factors, and the omission of important variables Puhan et al. (2009) . Openly discussing limitations is crucial. It upholds credibility and scientific integrity by demonstrating a commitment to ethical and transparent research practices Bunniss and Kelly (2010) ; Chasan-Taber (2014) ; Annesley (2010) ; Žydžiūnaitė (2018) . It also clarifies the scope of a study, supporting accurate interpretation, transferability, and reproducibility Ioannidis (2007) ; Eva and Lingard (2008) . In addition, it helps researchers avoid repeating the same shortcomings Escande et al. (2016) while creating opportunities to refine methods and guide future research Azher et al. (2025) . Despite these benefits, researchers are often reluctant to include limitations or articulate them in detail Ioannidis (2007) ; Ter Riet et al. (2013) . Concerns about the potential impact on publication chances and career progression Montori et al. (2004) can reinforce this tendency. Even when required to acknowledge limitations, as is now common in NLP/ML research, authors sometimes resort to generic or irrelevant statements that obscure the study’s real constraints Ross and Bibler Zaidi (2019) . Moreover, limitations may serve as a form of hedging , where findings are presented cautiously to avoid making definitive claims Hyland (1998) . This practice, while safer for authors, reduces the clarity and usefulness of the research. Failure to disclose limitations undermines the scientific process and misleads readers, reviewers, and policymakers, preventing recognition of constrained findings and potential biases Greener (2018) . Meanwhile, the volume of scientific publications has surged Bornmann et al. (2021) . These factors highlight the need for computational methods to study research limitations. However, progress in NLP toward automatic extraction, generation, and evaluation of limitations remains limited, largely due to the lack of standardized datasets, novel methods, and robust evaluation frameworks. This study takes a step toward closing this gap. Our contributions are as follows (see Figure 1 ): • Dataset creation . We build a dataset of research limitations by extracting them from papers and their reviews. By integrating author-reported and reviewer-identified limitations, this benchmark reduces self-reporting bias and provides a broader, more reliable resource for analyzing limitations and their impact on research. • Limitation generation . We design a novel RAG system to automatically generate limitations, offering a way to supplement papers with high-quality, context-aware limitation statements. • Evaluation framework . We introduce a new evaluation paradigm for generated limitations. Unlike traditional metrics (e.g., ROUGE Lin (2004) , BLEU Papineni (2001) , BERTScore Zhang et al. (2019) , MoverScore Zhao et al. (2019) ), which overemphasize common terms (e.g., bias, dataset, and generalizability), our framework leverages LLMs-as-judges for fine-grained, interpretable assessments and actionable error analysis. 2 Related Work Several studies have examined how limitations are reported in papers. Ioannidis (2007) found that only 17% of top-tier articles mentioned limitations, with just 1% doing so in abstracts. Similarly, Puhan et al. (2012) reported that 27% of biomedical papers lacked limitations, risking overestimation of research reliability. Goodman et al. (1994) noted that acknowledging limitations is often problematic in peer review. Few journals require discussing limitations Ioannidis (2007) , which can bias reviews and weaken scientific dialogue Horton (2002) , highlighting the need for greater transparency. Recent work has explored computational approaches to research limitations. Faizullah et al. (2024) proposed an LLM-chain pipeline to summarize and refine candidate limitations. Al Azher et al. (2024) integrated topic modeling with LLMs to derive structured limitation themes. Al Azher (2024) developed a graph-augmented LLM method for generating detailed limitation statements. Other studies address the shortcomings of visualizations by generating more meaningful captions for charts and graphs Al Azher and Alhoori (2024) . However, these studies are limited to ACL/EMNLP corpora and rely on author-stated limitations, and use metrics such as ROUGE and BERTScore that miss finer-grained contextual alignment. Concurrent to our work, Xu et al. (2025) also address these gaps by introducing LIMITGEN, a benchmark that incorporates human-written peer reviews to systematically evaluate how well LLMs identify limitations. Our framework complements this effort by additionally leveraging cited papers for broader context and introducing a novel limitation-level evaluation method to preserve granularity. Evaluating NLP outputs is essential for assessing quality, accuracy, and relevance. Traditional metrics like ROUGE and BLEU struggle with semantics, while BERTScore improves similarity but relies on references and lacks meaningful error analysis. Advances in large language models (LLMs) have opened new evaluation avenues Zheng et al. (2023) , from zero-shot and in-context learning Wei et al. (2022) to specialized approaches such as GPTScore Fu et al. (2023) , TIGERScore Jiang et al. (2023) , and PandaLM Wang et al. (2023) . Other methods include AttrScore Yue et al. (2023) , which checks factual support, and SummacConv Laban et al. (2022) , which filters low-entailment sentences. Despite their promise, LLM-based evaluations face issues such as positioning bias, where input order can shift results. We address this by randomizing order and retaining stable outputs. More broadly, our evaluation advances beyond prior work by combining granularity-aware scoring, topic-level agreement, and LLMs-as-judges. Taken together, prior research shows both the need and the opportunity for a more systematic treatment of research limitations. Building on these insights, our work unifies dataset construction, limitation generation, and evaluation into a single framework, laying the foundation for more transparent and reproducible analysis of limitations. 3 Limitation Extraction & Evaluation 3.1 Dataset of Extracted Limitations Granularity. A key challenge in building a dataset of research limitations is defining the appropriate level of granularity. Should a limitation be captured as a single