DEVELOPMENT AND ASSESSMENT OF THE PERFORMANCE OF A LARGE LANGUAGE MODEL FOR ADMINISTERING THE SHORT BLESSED TEST

Rafeeul Jaman, Sarah Nessen, Michael Adjei-Poku, Joella Byerley, Olivia Sailors, Jason Karlawish, Kyra O’Brien, Ari Friedman · 2024

Read DEVELOPMENT AND ASSESSMENT OF THE PERFORMANCE OF A LARGE LANGUAGE MODEL FOR ADMINISTERING THE SHORT BLESSED TEST by Rafeeul Jaman, Sarah Nessen, Michael Adjei-Poku, Joella Byerley, Olivia Sailors, Jason Karlawish, Kyra O’Brien, Ari Friedman (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/pmc-11691942.txt.

Description

Rafeeul Jaman and co-authors test whether a language model can give a standard memory screening test in this 2024 study. Screening whole hospital populations for mild impairment could widen access to new therapies, but staff time is short. Language models might help, though they can invent facts and slip on arithmetic.

The team tuned a leading model to ask the test questions and report item scores, leaving the adding to simple math. A researcher then role-played patients across a grid of intended scores. The model's total matched the intended score in just over half of the runs, and nearly all errors were small. It never flagged a healthy case as impaired. The authors see a support role, with humans confirming results.

Catalog record

CREATORRafeeul Jaman, Sarah Nessen, Michael Adjei-Poku, Joella Byerley, Olivia Sailors, Jason Karlawish, Kyra O’Brien, Ari Friedman
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE2 KB
VERSION1.0
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC11691942

Browse the full library · About the Tower of Babel