Explainable cognitive decline detection in free dialogues with a Machine Learning approach based on pre-trained Large Language Models

Francisco de Arriba-Pérez, Silvia García-Méndez, Javier Otero-Mosquera, Francisco J. González-Castaño · 2024

Read Explainable cognitive decline detection in free dialogues with a Machine Learning approach based on pre-trained Large Language Models by Francisco de Arriba-Pérez, Silvia García-Méndez, Javier Otero-Mosquera, Francisco J. González-Castaño (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02036.txt.

Description

Abstract Cognitive and neurological impairments are very common, but only a small proportion of affected individuals are diagnosed and treated, partly because of the high costs associated with frequent screening. Detecting pre-illness stages and analyzing the progression of neurological disorders through effective and efficient intelligent systems can be beneficial for timely diagnosis and early intervention.

Catalog record

CREATORFrancisco de Arriba-Pérez, Silvia García-Méndez, Javier Otero-Mosquera, Francisco J. González-Castaño
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2411-02036

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Explainable cognitive decline detection in free dialogues with a Machine Learning approach based on pre-trained Large Language Models Francisco de Arriba-Pérez, Silvia García-Méndez, Javier Otero-Mosquera, Francisco J. González-Castaño Abstract Cognitive and neurological impairments are very common, but only a small proportion of affected individuals are diagnosed and treated, partly because of the high costs associated with frequent screening. Detecting pre-illness stages and analyzing the progression of neurological disorders through effective and efficient intelligent systems can be beneficial for timely diagnosis and early intervention. We propose using Large Language Models to extract features from free dialogues to detect cognitive decline. These features comprise high-level reasoning content-independent features (such as comprehension, decreased awareness, increased distraction, and memory problems). Our solution comprises ( i ) preprocessing, ( ii ) feature engineering via Natural Language Processing techniques and prompt engineering, ( iii ) feature analysis and selection to optimize performance, and ( iv ) classification, supported by automatic explainability. We also explore how to improve Chat gpt ’s direct cognitive impairment prediction capabilities using the best features in our models. Evaluation metrics obtained endorse the effectiveness of a mixed approach combining feature extraction with Chat gpt and a specialized Machine Learning model to detect cognitive decline within free-form conversational dialogues with older adults. Ultimately, our work may facilitate the development of an inexpensive, non-invasive, and rapid means of detecting and explaining cognitive decline. 2023 This version of the article has been accepted for publication, after peer review (when applicable) but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/s10489-024-05808-0. keywords Artificial Intelligence, cognitive decline, explainability, Large Language Models, Machine Learning, Natural Language Processing 1 Introduction Progressive neurological disorders ( e.g. , Alzheimer’s disease) affect 40 million people worldwide [ 1 ] and are a common cause of death [ 2 , 3 ] . However, only 25\text{\,}\mathrm{\%} of affected people receive a diagnosis. There are multiple reasons for this, including stigma and a lack of awareness and resources [ 4 , 5 ] . The number of older adults with Alzheimer’s disease is predicted to rise to 150 million by 2050 [ 6 ] . Consequently, detecting pre-illness stages and analyzing the progression of neurological disorders using cost-effective and efficient intelligent systems is crucial to ensure timely diagnosis, risk assessment, and early intervention [ 3 , 7 ] . Numerous cognitive tests ( ct , e.g. , Alzheimer’s Disease Assessment Scale-Cognition - adasc og, Cognitive Dementia Rating - cdr , Mini-Mental State Examination - mmse , Montreal Cognitive Assessment - Mo ca , etc.) are currently used to diagnose and monitor neurological disorders, but they need to be applied manually and are therefore costly [ 5 , 8 , 9 ] . These tests can be automated with the help of Artificial Intelligence ( ai ), enabling more frequent screening of target populations [ 10 , 11 ] and the conduct of longitudinal studies. The proposal described in this work consists of an automatic, continuous evaluation of cognitive performance based on engaging dialogues established with end users. Engagement fosters evaluation over time, which is of interest to longitudinal studies. The dialogues are supported by advanced ai techniques. ai -based solutions for clinical assessment purposes is a fast-growing research field, and numerous studies have already analyzed applications in the fields of progressive neurological disorders and dementia [ 12 , 13 , 14 , 15 , 16 ] . Machine Learning ( ml ) models [ 1 , 17 , 18 , 19 ] ( e.g. , Convolutional Neural Networks - cnn [ 20 ] ) and Natural Language Processing ( nlp ) techniques [ 21 , 22 , 15 , 23 ] have been applied to textual and voice data. These techniques can effectively predict cognitive impairment and content-dependent and context-independent features can be leveraged to improve the predictive performance of ml models in this area. Among the multiple bio-markers available for the study of cognitive decline [ 24 , 25 , 26 ] , language acquisition is an inexpensive, non-invasive, and readily accessible tool [ 9 ] . However, language conceals large volumes of information within complex relationships that can be difficult to decipher. Large Language Models ( llm s) are better equipped to navigate and process complex information and have therefore received increasing attention in the medical field [ 1 , 27 , 28 ] . llm s have been used to analyze images and prescribe medical treatments and have also proven capable of passing medical accreditation exams [ 29 ] . However, personalized medical treatment recommendations provided by llm s remain unreliable [ 30 ] . llm s have broader language generation and understanding capabilities [ 9 , 31 ] , compared to domain-dependent models, when adequate prompt engineering is applied [ 32 ] . One noteworthy recent example of llm enhanced text generation capabilities is gpt-4 [ 33 ] . This llm is used by Chat gpt [ 34 ] to generate coherent dialogues with human users. However, there is still limited research on whether llm s are better than niche solutions based on highly focused, context- or domain-dependent ml models, particularly in the clinical field [ 33 ] . Our approach combines both approaches. It employs an llm to create a friendly conversational assistant environment. The dialogue is augmented by additional services ( e.g. , weather reports, and medication reminders) to generate interest among the end users and enable a transparent longitudinal evaluation of cognitive decline. This llm is also used to generate high-level reasoning and content-independent side features for a specialized ml model. This combined usage of an llm and ml model for conducting explainable cognitive evaluation is an entirely novel contribution. In general, establishing trust in ai is necessary to fight the common perception that models are black boxes, leaving end users and developers in the dark about their decision-making processes [ 35 ] . This issue is especially relevant in medical practice, particularly in personalized treatments [ 9 ] . Explainable ai ( xai ) [ 36 ] exploits the intrinsic interpretability of certain ml algorithms ( e.g. , tree-based models such as Random Forest - rf ), or implements methods to bypass the opacity of non-interpretable models [ 37 ] . xai techniques include feature importance [ 38 ] , counterfactual explanations [ 39 ] , natural language descriptions [ 40 ] , and visual representations [ 41 ] . Explainability in llm models is far from being solved. Our proposal combines llm s with explainable ml classifiers to generate an explanation of the predictions that caregivers or healthcare experts will be able to understand. Summing up, the main contribution of this work is a hybrid ml solution for assessing cognitive state, which combines context-dependent features with an llm for generating high-level reasoning, context-independent features. The latter provides high-level reasoning on the behavior of a user who dialogues with a conversational assistant on a leisure topic. Moreover, these features greatly enhance the explainability of the system decisions about cognitive state compared to rigid content-dependent features. The rest of this paper is organized as follows. Section 2 reviews the relevant competing works

Read the complete text