LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues

Amir Ivry, Shinji Watanabe · 2026

Read LALM-as-a-Judge: Benchmarking Large Audio-Language Models for Safety Evaluation in Multi-Turn Spoken Dialogues by Amir Ivry, Shinji Watanabe (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-04796.txt.

Description

Amir Ivry and Shinji Watanabe benchmark audio-language models as safety judges for spoken dialogue in a 2026 study. Evaluating harmful content in speech has stayed text-centric, missing tone of voice and errors that transcription introduces.

Their benchmark contains many thousands of multi-turn spoken dialogues, each with one localized unsafe turn drawn from several harm categories and severity levels. Six models, open and closed, were tested in text-only, audio-only, and combined setups. The authors measured sensitivity, severity ordering, and bias by turn position. Audio added evidence beyond the words themselves. Combined setups helped unevenly, with some models anchored to text, others balanced or conservative. The paper closes with practical guidance for choosing models, modalities, and prompts.

Catalog record

CREATORAmir Ivry, Shinji Watanabe
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE73 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-04796

Browse the full library · About the Tower of Babel