From Models to Microtheories: Distilling a Model’s Topical Knowledge for Grounded Question Answering
Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Samuel Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark · 2024
Read From Models to Microtheories: Distilling a Model’s Topical Knowledge for Grounded Question Answering by Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Samuel Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17701.txt.
Description
Nathaniel Weir and co-authors extract a language model's knowledge on a topic into a compact, readable theory, in a 2024 paper. Reasoning traces explain single answers, but they hide what the model overall believes about a subject. The team calls the extracted set of core statements a microtheory.
They first fill a store with model-written sentences that support answers to training questions, then distill it down to a concise, general, non-redundant core. Added to a general corpus like Wikipedia, these mini-theories supplied missing key facts. Models grounded up to eight percent more answers in verifiable text, and grounded answers grew up to eight percent more accurate. Human judges in a medical test found the distilled versions packed with far more of the facts that matter. The authors also define a coverage score for judging how complete a microtheory is.
Catalog record
| CREATOR | Nathaniel Weir, Bhavana Dalvi Mishra, Orion Weller, Oyvind Tafjord, Samuel Hornstein, Alexander Sabol, Peter Jansen, Benjamin Van Durme, Peter Clark |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 76 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17701