From Models to Microtheories: Distilling a Model’s Topical Knowledge for Grounded Question Answering

Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Samuel Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark · 2024

Read From Models to Microtheories: Distilling a Model’s Topical Knowledge for Grounded Question Answering by Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Samuel Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17701.txt.

Description

Nathaniel Weir and co-authors extract a language model's knowledge on a topic into a compact, readable theory, in a 2024 paper. Reasoning traces explain single answers, but they hide what the model overall believes about a subject. The team calls the extracted set of core statements a microtheory.

They first fill a store with model-written sentences that support answers to training questions, then distill it down to a concise, general, non-redundant core. Added to a general corpus like Wikipedia, these mini-theories supplied missing key facts. Models grounded up to eight percent more answers in verifiable text, and grounded answers grew up to eight percent more accurate. Human judges in a medical test found the distilled versions packed with far more of the facts that matter. The authors also define a coverage score for judging how complete a microtheory is.

Catalog record

CREATORNathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Samuel Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-17701

Browse the full library · About the Tower of Babel