Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge

Yunna Cai · 2025

Read Exploring Safety Alignment Evaluation of LLMs in Chinese Mental Health Dialogues via LLM-as-Judge by Yunna Cai (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-08236.txt.

Description

Yunna Cai and co-authors ask in 2025 how to judge whether a language model replies safely in Chinese mental health chats. Such chats are high risk, and there is no set of gold standard replies to check against. What counts as safe must come from expert rules for care in a crisis.

The team built a test set from real Chinese chats about mental health. A second model acts as judge, scoring each reply against expert defined steps that reflect sound crisis care. Scores are simple pass or fail points for each reply. The method needs no fixed answer key, which suits a field where the right reply depends on context. Tests like this could help teams check chat tools before they reach people in distress.

Catalog record

CREATORYunna Cai
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE60 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-08236

Browse the full library · About the Tower of Babel