Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses
Khizar Hussain, Bradley A. Malin, Zhijun Yin, Susannah Leigh Rose, Murat Kantarcioglu · 2026
Read Blending Human and LLM Expertise to Detect Hallucinations and Omissions in Mental Health Chatbot Responses by Khizar Hussain, Bradley A. Malin, Zhijun Yin, Susannah Leigh Rose, Murat Kantarcioglu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-06216.txt.
Description
Hussain and four colleagues tackle hallucination and omission detection in mental health chatbot replies, where judges fail badly. Leading judges reach only 52 percent accuracy on counseling data, with some checks near zero recall, because they miss subtle language and therapy patterns. The team blends human expertise with models, pulling domain-informed features across five areas: logical consistency, entity checks, factual accuracy, language uncertainty, and professional fit. Standard machine learning on those features reaches 0.849 F1 on a public benchmark, beating black-box judging on trust and clarity.
Catalog record
| CREATOR | Khizar Hussain, Bradley A. Malin, Zhijun Yin, Susannah Leigh Rose, Murat Kantarcioglu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 58685 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.06216