The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input
Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, · 2025
Read The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input by Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-03200.txt.
Introduction
Jacovi and co-authors launch a public test of whether AI answers stick to their source text in a 2025 paper. Their benchmark, FACTS Grounding, hands models a full document and a request. Answers must fill the request using only the document, even for long inputs. Judge models grade the replies in two steps.
Replies that miss the request are thrown out first. The rest pass only if every claim rests on the document. Several judge models vote to blunt single-judge bias. Public and private splits let outsiders join while guarding the test. The board lives online and will be kept current, the team says.
Catalog record
| CREATOR | Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 41 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-03200
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein, Dipanjan Das Abstract We introduce FACTS Grounding , an online leaderboard and associated benchmark that evaluates language models’ ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at https://www.kaggle.com/facts-leaderboard . 1 Introduction Factuality is one of the most challenging aspects of Large Language Models (LLMs), referring to a model’s ability to generate factually accurate responses in information-seeking scenarios. Commonly, this area of research can be divided into two distinct scenarios: (1) factuality with respect to given context , such as a user request and grounding documents, such that the model response is fully grounded in the input (by this, we imply that a model response has the highest degree of faithfulness to given context as defined by Rashkin et al. 2023 ), and (2) factuality with respect to external sources and general world knowledge ( Tang et al. 2024 , cf. Rashkin et al. 2023 ; Pan et al. 2023 ; Zhao et al. 2024b ). There may be subtle use cases that fall in the middle, but most existing literature in the factuality area focus on behaviors that map to these two distinct scenarios. Summarization is a narrow but important example of the first scenario–a summary’s various claims should be accurate with respect to the source documents that are being summarized ( Bishop et al. 2023 ; Krishna et al. 2023 ) . Generating short-form, factually accurate answers to factoid questions from an LLM’s parametric knowledge is an example use case of the second scenario, as recently discussed by Wei et al. 2024a . Ensuring factual accuracy while generating LLM responses is challenging. The principal challenges in LLM factuality are modeling (i.e., architecture, training, and inference) and measurement (i.e., evaluation methodology, data and metrics). On the modeling front, LLMs are inherently difficult to optimize for this goal: typically, LLM pretraining is optimized to predict the next token given previous tokens in the text. While this objective may teach models salient world knowledge, it does not directly optimize the model towards the various factuality scenarios, instead encouraging the model to generate generally plausible text. Furthermore, critically, recent research suggests that the training process (regardless of specific models or training data) inherently admits non-factual text generation ( Kalai and Vempala 2024 ) . Subsequent post-training, via supervised finetuning and reinforcement learning, can steer the model towards improved factuality ( Lan et al. 2023 ; Roit et al. 2023 ; Huang and Chen 2024 ) . Other mitigation proposals include inference-time methods, such as prompting or model state interpretability ( Lee et al. 2024 ; Su et al. 2024 ) . However, enhancing factuality through these methods can compromise other desirable attributes, such as creativity and novelty, which are also optimized during these stages. This creates a delicate balance, making it difficult to simultaneously achieve all desired outcomes ( Roit et al. 2023 ) . Due to the above, factuality is expected to remain a research challenge for the foreseeable future. In this work, we focus on the challenge of measuring progress in factuality. Measurement is in itself difficult, due to the different model behaviors that pertain to the aforementioned factuality scenarios. Some settings are more difficult than others: in long-form generation tasks, the subject of our benchmark, each claim in the models’ responses should be thoroughly inspected for their accuracy. This is in contrast to tasks that require measuring the factuality of short responses to questions against ground truth ( Wei et al. 2024a , e.g.,) . To complicate matters further, long-form generation settings are in themselves varied and numerous. For example, some benchmarks focus on long-form generation in the setting of grounding to external sources ( Wei et al. 2024b ; Zhu et al. 2024 ; Zhao et al. 2024a , e.g.,) while others focus on a particular task or domain, such as summarization to news or biomedical articles ( Vectara 2024 ; Ramprasad et al. 2024 , e.g.,) . Here, we investigate a benchmark that focuses on scenario (1)–particularly, the measurement of response factuality with respect to a provided context document of length up to 32k tokens, given varied user requests. This setting requires the model to synthesize information derivable from the document while directly addressing the request. While encompassing summarization as a key use case, it extends to a broader range of requests, including fact finding, analyzing and comparing information, and so forth, while being fully grounded to the input document. We believe that this benchmark fills a gap in evaluating a wider variety of model behaviors pertaining to factuality, in comparison to benchmarks that focus on narrower use cases, e.g. summarization alone ( Vectara 2024 ) . (Note however that we do not capture scenario (2) in this work.) Table 1 : Examples from FACTS Grounding (Public). System Instruction Context Document Description Context Tokens User Request Answer the question using only the information provided in the context. Do not rely on external knowledge or sources. The development and deployment of an autonomous robotic system designed to clean skyscraper windows, highlighting its technological advancements, safety implications, and potential impact on the window-washing industry. \sim 1.1k My sister and her dog live in NYC. I’ve visited there and have always been fascinated with their tall buildings. Then I thought…someone has to clean those! Then next thing you know, window washing robotos popped up on my feed. How do these robots work? Also what does this mean for the people who do those jobs? Provide a response based solely on the information provided in the prompt. External sources and prior knowledge must not be used. Legal interpretations and effects of the medical marijuana appropriations rider on federal marijuana prosecutions, focusing on differing circuit court approaches to determining compliance with state medical marijuana laws. \sim 1.6k What did the first circuit conclude? This task requires you to answer questions based solely on the information provided in the prompt. You are not allowed to