Assessing the Limitations of Large Language Models in Clinical Fact Decomposition
Monica Munnangi*, Akshay Swaminathan*, Jason Alan Fries*, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah · 2024
Read Assessing the Limitations of Large Language Models in Clinical Fact Decomposition by Monica Munnangi*, Akshay Swaminathan*, Jason Alan Fries*, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12422.txt.
Description
Monica Munnangi and co-authors present, in a 2024 paper, a data set for testing how well language models break clinic notes into single facts. Checking facts is key if such models are to be used in care with care. The split step turns a dense note into short lines, each with one fact, so each can be checked on its own. Clinic notes are dense with terms and come in many forms, which makes the split hard.
Their set, FactEHR, covers 2,168 notes of four types from three hospital systems. It holds near one million pairs that test if one line follows from a note. Clinicians helped judge the output. One model, Gemini-1.5-Flash, made sound facts in a steady way. A smaller model, Llama-3 8B, made fewer and less steady facts. The set gives builders a hard, shared test.
Catalog record
| CREATOR | Monica Munnangi*, Akshay Swaminathan*, Jason Alan Fries*, Jenelle Jindal, Sanjana Narayanan, Ivan Lopez, Lucia Tu, Philip Chung, Jesutofunmi A. Omiye, Mehr Kashyap, Nigam Shah |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 44 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12422