: Organized Literature Synthesis with Attribution
Amanpreet Singh, Joseph Chee Chang *, Chloe Anastasiades *, Dany Haddad *, Aakanksha Naik Amber Tanaka Angele Zamarron Cecile Nguyen Jena D. Hwang · 2025
Read : Organized Literature Synthesis with Attribution by Amanpreet Singh, Joseph Chee Chang *, Chloe Anastasiades *, Dany Haddad *, Aakanksha Naik Amber Tanaka Angele Zamarron Cecile Nguyen Jena D. Hwang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-10861.txt.
Introduction
Singh and a large team of co-authors introduce Ai2 Scholar QA, a free web application that answers scientific questions from the research literature. Systems that search papers and then write cited answers have grown powerful, but many leading ones are costly and closed, which shuts out smaller research groups.
This team released the whole pipeline as an open-source Python package with a public web app, open paper indexes, and downloadable datasets. They describe the design in detail and test the choices behind it. On a recent scientific question-answering benchmark, the system outperformed its competitors. The authors frame openness itself as the contribution: anyone can inspect, reuse, and improve the tool.
Catalog record
| CREATOR | Amanpreet Singh, Joseph Chee Chang *, Chloe Anastasiades *, Dany Haddad *, Aakanksha Naik Amber Tanaka Angele Zamarron Cecile Nguyen Jena D. Hwang |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2504-10861
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
: Organized Literature Synthesis with Attribution Amanpreet Singh, Joseph Chee Chang *, Chloe Anastasiades *, Dany Haddad *, Aakanksha Naik Amber Tanaka Angele Zamarron Cecile Nguyen Jena D. Hwang Abstract Retrieval-augmented generation is increasingly effective in answering scientific questions from literature, but many state-of-the-art systems are expensive and closed-source. We introduce Ai2 Scholar QA, a free online scientific question answering application. To facilitate research, we make our entire pipeline public: as a customizable open-source Python package 1 1 1 We use closed state-of-the-art LLMs. and interactive web app, along with paper indexes accessible through public APIs and downloadable datasets. We describe our system in detail and present experiments analyzing its key design decisions. In an evaluation on a recent scientific QA benchmark, we find that Ai2 Scholar QA outperforms competing systems. qa.allen.ai allenai/ai2-scholarqa-lib Demo Video Python Package 1 Introduction Long-form scientific question answering systems use retrieval-augmented generation (RAG) Lewis et al. (2020) over scientific literature to answer complex questions. These systems produce responses that bring together relevant insights from dozens of papers to help users rapidly learn about a body of scientific work. Examples are OpenScholar Asai et al. (2024) , Elicit, Consensus, and others § 5 . Most of these systems are expensive to use and closed source, relying on models, workflows, and retrieval solutions not shared publicly. These issues create barriers for researchers who wish to study or build on the work. In response, we introduce Ai2 Scholar QA, a free-to-use scientific QA system ( qa.allen.ai ), and share our key components as open source software and public APIs. Scholar QA follows a multi-stage pipeline ( Figure 1 ) that starts by querying paper indexes: one from Semantic Scholar with over 100M abstracts, and a new index that we introduce in this work containing 11.7M full-text scientific papers. The pipeline then re-ranks the retrieved passages with a cross-encoder, and finally prompts a Large Language Model (LLM) to filter, cluster, and synthesize the passages into an answer. The final answer is presented to the user in a report with expandable sections of prose, bulleted lists, and tables. Claims in the answer are supported by citations, which can be clicked to reveal the cited paper’s title and authors (with links to their corresponding Semantic Scholar pages), and in many cases relevant excerpt(s) from the paper, allowing for quick verification of the claim. Figure 1: Scholar QA Pipeline Overview The system is based on open source code, enabling the community to reproduce and build on it. We release the code for our pipeline, prompting workflow and Web application. The retrieval indexes, including the new full text search index, are available as Semantic Scholar APIs and dataset downloads, and are continually updated with new articles Kinney et al. (2023) . Together, these resources can be combined with any generative LLM API to power a complete long-form scientific QA application. Our production system currently uses Anthropic’s Claude 3.7 ( Anthropic, 2024 ) . We present analyses that justify key design decisions in our architecture in § 4 . Our choice of retrieval models and configuration is informed by evaluation over a collection of real and synthetic user queries and accompanying passages judged for relevance by a LLM, both of which we release publicly. We compare Scholar QA’s answers against several baselines, demonstrating that it achieves state-of-the-art performance on the ScholarQA-CS benchmark ( Asai et al., 2024 ) . Finally, we discuss the reception of Scholar QA by users. The strong majority (85%) of user feedback is positive, and the reported issues suggest important improvements for future work. 2 Pipeline The Scholar QA architecture (Figure 1 ) has three primary components: 1) retrieval to identify relevant passages from a corpus of scientific literature; 2) a neural cross-encoder that re-ranks the passages to select the most relevant top-k; and 3) multi-step LLM generation to process the passages into a comprehensive report. Next, we describe each component of the pipeline in detail. Query Validation. Prior to processing a query, we employ OpenAI’s omni-moderation-latest 2 2 2 https://platform.openai.com/docs/guides/moderation model for safeguarding against potentially harmful content and return appropriate error messages. 2.1 Retrieval We use the Semantic Scholar API ( Kinney et al., 2023 ) for retrieval, specifically its endpoint for keyword search over paper abstracts, and our new endpoint for querying snippets from open-access papers. A query decomposer re-formulates the user query for each endpoint and retrieves up to 256 snippets and 20 abstracts. These texts are referred to as "passages" below. Query Decomposer. The two retrieval endpoints differ in their effective query formats (one targets keyword and the other semantic queries) and filtering of results based on the user’s preferences for paper metadata (paper year, venue, field of study). In our query decomposition step, an LLM is prompted to re-format the user query into paraphrases appropriate for each endpoint, and to extract the user’s requested settings for the metadata filters. We use the outputs of this step for retrieval. Search APIs. The Semantic Scholar keyword search API is described in Kinney et al. (2023) . We introduce a new /snippet/search endpoint , which searches over a corpus of passages extracted from S2ORC ( Lo et al., 2020 ) , loaded into a Vespa cluster with papers and passages. Papers include metadata for filtering. Passages are derived from a paper’s title, abstract, or body and can be filtered at the paper level. The index includes 11.7M full-text papers across the fields of study listed here , and a total of 285.6M passages. Each passage is limited to 480 tokens and truncated at sentence and section boundaries where possible, having an overlap of one sentence (up to 64 tokens) with the preceding and following passages. Passage text is embedded with mxbai-embed-large-v1 ( Lee et al., 2024 ) with binary quantization, and placed into a dense (approximate nearest neighbor) index, as well as a traditional sparse keyword index. We first retrieve a union of embedding and keyword-based matches, applying any specified filters. The filtered results are ranked with a weighted sum of embedding similarity and bm25 scores. 2.2 Reranking The passages obtained from the retrieval step are subsequently passed to a neural re-ranker and the top 50 results are retained. The re-ranker is a cross-encoder that encodes both the query and a candidate document simultaneously and outputs a relevance score used to rank the documents. We selected mxbai-rerank-large-v1 ( Shakir et al., 2024 ) based on the results in § 4.2 and host it on Modal with a single NVIDIA L40S GPU. 2.3 Multi-step Generation The generation phase employs a three-step approach: first, the retrieved passages are processed to extract more precise quotes relevant to the query; second, the quotes are thematically clustered into separate sections appropriate for the answer; finally, a controlled generation process composes the final report one section at a time, synthesizing the quotes assigned to that section. Quote extraction. Passages from the retrieval stage can be lengthy and may contain extraneous information not useful for answering the user query ( Asai et al., 2023 ) . The quote extraction stage aims to select only the most relevant quotes from the passages to improve the precision of the answer. We instruct an LLM to extract verbatim