Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models
Emma Croxford 2 , Yanjun Gao PhD 5 , Nicholas Pellegrino 7 , Karen K. Wong MD 7,8 , Graham Wills PhD 8 , Elliot First 7 , Miranda Schnier 7 , Kyle Burton MD 8 , Cris G. Ebby MD, MS 4 , Jillian Gorskic MD 8 , Matthew Kalscheur MD 1,8 , Samy Khalil MD 8 , Marie Pisani BS 1 , Tyler Rubeor MD 8 , Peter · 2025
Read Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Models by Emma Croxford 2 , Yanjun Gao PhD 5 , Nicholas Pellegrino 7 , Karen K. Wong MD 7,8 , Graham Wills PhD 8 , Elliot First 7 , Miranda Schnier 7 , Kyle Burton MD 8 , Cris G. Ebby MD, MS 4 , Jillian Gorskic MD 8 , Matthew Kalscheur MD 1,8 , Samy Khalil MD 8 , Marie Pisani BS 1 , Tyler Rubeor MD 8 , Peter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08977.txt.
Description
Emma Croxford and co-authors wrote a 2025 paper on a grading tool for AI-written summaries of medical notes. Language models now draft summaries inside health records, and hospitals need a checked ruler before trusting them. Older quality forms were not built for machine text and were rarely tested on real records.
Their nine-item instrument was built and tested on summaries made by several models from real patient files across specialties. Human graders used the form, and the team checked its consistency, structure, and agreement between graders. A panel process refined the items. A validated ruler lets hospitals compare models fairly and spot weak summaries before they reach a chart.
Catalog record
| CREATOR | Emma Croxford 2 , Yanjun Gao PhD 5 , Nicholas Pellegrino 7 , Karen K. Wong MD 7,8 , Graham Wills PhD 8 , Elliot First 7 , Miranda Schnier 7 , Kyle Burton MD 8 , Cris G. Ebby MD, MS 4 , Jillian Gorskic MD 8 , Matthew Kalscheur MD 1,8 , Samy Khalil MD 8 , Marie Pisani BS 1 , Tyler Rubeor MD 8 , Peter |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-08977