Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization
Mengjie, Lianbo, Yusuke, Hao, Yuan, Roman, Yui · 2026
Read Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization by Mengjie, Lianbo, Yusuke, Hao, Yuan, Roman, Yui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-12565.txt.
Introduction
Mengjie Zhao and co-authors adapt Japanese speech language models so their text sounds right when read aloud. Models trained on written text tend to keep written style, which fits speech synthesis poorly. In Japanese, spoken and written forms differ in politeness marks, sentence endings, and structure. The team uses direct preference optimization to favor short, conversational output. They also built SpokenElyza, a test set checked by native speakers, where their method gained a lot while keeping written-task scores largely intact.
Catalog record
| CREATOR | Mengjie, Lianbo, Yusuke, Hao, Yuan, Roman, Yui |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 23856 |
| LICENSE | http://creativecommons.org/licenses/by-sa/4.0/ |
| RIGHTS | CC BY SA |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.12565
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization Mengjie, Lianbo, Yusuke, Hao, Yuan, Roman, Yui Abstract SpeechLLMs typically combine ASR-trained encoders with text-based LLM backbones, leading them to inherit written-style output patterns unsuitable for text-to-speech synthesis. This mismatch is particularly pronounced in Japanese, where spoken and written registers differ substantially in politeness markers, sentence-final particles, and syntactic complexity. We propose a preference-based alignment approach to adapt Japanese SpeechLLMs for speech-worthy outputs: text that is concise, conversational, and readily synthesized as natural speech. To rigorously evaluate this task, we introduce SpokenElyza , a benchmark for Japanese speech-worthiness derived from ELYZA-tasks-100 with auditory verification by native experts. Experiments show that our approach achieves substantial improvement on SpokenElyza while largely preserving performance on the original written-style evaluation. We will release SpokenElyza to support future research on Japanese spoken dialog systems. Zhao Liu Fujita Shi Gao Koshkin Sudo keywords SpeechLLM, speech-worthy alignment, direct preference optimization, spoken dialog, Japanese † † address: SB Intuitions, Tokyo, Japan 1 Introduction Extending large language models (LLMs) with auditory capabilities has enabled numerous real-world applications, including automatic speech recognition (ASR; [ 16 , 22 , 19 ] ), sound event understanding [ 11 , 10 ] , and music comprehension [ 9 , 23 , 15 ] . These auditory LLMs process audio inputs and generate textual responses, bridging the gap between audio/speech and language understanding. However, SpeechLLMs like Qwen2-audio [ 5 ] are typically trained to respond in written text rather than spoken language [ 3 ] . This creates a fundamental mismatch for speech-to-speech applications, where the model’s output must be synthesized into natural-sounding audio via text-to-speech (TTS) systems. LLMs’ written-style outputs often incorporate markdown formatting, bullet points, and complex sentence structures, which are poorly suited for auditory consumption. To address this, [ 3 ] defined speech-worthy responses : text that is concise, avoids non-verbalizable artifacts (e.g., bullet points, markdown), and prioritizes natural conversational flow for auditory comprehension. Cascaded systems combining SpeechLLMs with TTS remain the dominant paradigm for speech-to-speech interaction. While end-to-end models such as SLAM-Omni [ 1 ] and Llama-Omni [ 8 ] can generate speech directly, they often suffer from catastrophic forgetting and underperform their cascaded counterparts, e.g., on VoiceBench [ 2 ] . Consequently, producing speech-worthy text from SpeechLLMs remains a critical intermediate step for high-quality spoken dialog systems. Recent work has explored aligning text-based LLMs to produce speech-worthy outputs [ 3 , 21 ] . However, these approaches focus exclusively on text LLMs, leaving the growing ecosystem of SpeechLLMs unexplored. In this work, we address the challenge of aligning SpeechLLMs to generate speech-worthy outputs . SpeechLLMs are typically trained on ASR transcripts or text-based instruction data [ 4 ] , resulting in verbose, structured outputs optimized for reading rather than listening [ 21 ] . We focus specifically on Japanese which has pronounced stylistic divergence between written and spoken registers. Natural spoken Japanese requires polite predicates, sentence-final particles signaling speaker intent, and simpler syntactic structures. SpeechLLMs producing grammatically correct written Japanese may nonetheless sound robotic, overly dense, or socially inappropriate when synthesized as speech. These pronounced divergences make Japanese an ideal testbed for studying speech-worthy alignment. To the best of our knowledge, no evaluation resources currently exist for Japanese speech-worthy dialog. We address this gap by constructing SpokenElyza , a carefully curated benchmark derived from the widely-used ELYZA-tasks-100 dataset [ 18 ] . Our contributions are as follows: (1) We apply preference-based alignment to Japanese SpeechLLMs and demonstrate substantial improvement on speech-worthiness while largely preserving written-style performance. (2) To enable this evaluation, we introduce SpokenElyza, the first benchmark for speech-worthy Japanese spoken dialog, constructed through modality filtering, style transfer, and auditory verification by native experts. We will release SpokenElyza to facilitate future research in Japanese spoken dialog evaluation 1 1 1 https://huggingface.co/datasets/sbintuitions/voicebench-ja . Figure 1: Overview of our speech-worthy alignment. SpeechLLM responses are aligned from written style (verbose, symbol-heavy) to spoken style (concise, conversational) for auditory comprehension and TTS synthesis. English translations at right; non-speech-worthy elements are highlighted in red. 2 Method 2.1 Model Architecture Our model follows a widely-used SpeechLLM architecture similar to Qwen2-Audio [ 5 ] , where a shallow projector layer connects a speech encoder (from Whisper-large [ 16 ] ) with an LLM. The LLM consumes projected representations, optionally augmented with text embeddings (e.g., of system prompts), to generate outputs. We use Sarashina-7B 2 2 2 We use our internal weights; similar models are at https://huggingface.co/sbintuitions/sarashina2-7b as our base LLM to utilize its strong Japanese language capabilities. 2.2 Preference-Based Alignment Our goal is to adapt SpeechLLMs to produce speech-worthy outputs while preserving their instruction-following capabilities. We frame this as a preference optimization problem, where speech-worthy responses are preferred over written-style responses. Direct Preference Optimization (DPO). We employ DPO [ 17 ] to directly optimize a policy to satisfy the preference on speech-worthy without requiring a separate reward model. Given preference pairs (y_{w},y_{l}) where y_{w} is the speech-worthy (preferred) response and y_{l} is the written-style (dispreferred) response, DPO minimizes: \begin{split}\mathcal{L}_{\text{DPO}}(\pi_{\theta};\pi_{\text{ref}})&=-\mathbb{E}_{(x,y_{w},y_{l})\sim\mathcal{D}}\bigg[\\ &\hskip 8.50012pt\log\sigma\left(\beta\log\frac{\pi_{\theta}(y_{w}\mid x)}{\pi_{\text{ref}}(y_{w}\mid x)}-\beta\log\frac{\pi_{\theta}(y_{l}\mid x)}{\pi_{\text{ref}}(y_{l}\mid x)}\right)\bigg]\end{split} (1) where \pi_{\theta} is the policy being optimized, \pi_{\text{ref}} is the reference model (our pretrained checkpoint), \sigma is the logistic sigmoid function, and \beta controls the strength of the KL divergence constraint. Intuitively, DPO increases the relative likelihood of preferred responses while staying close to the reference distribution to prevent degeneration, in our case, deterioration on instruction following ability. Combined DPO + SFT Training. We combine DPO with supervised fine-tuning (SFT) on chosen responses, and the SFT loss directly maximizes the likelihood of speech-worthy responses: \mathcal{L}_{\text{SFT}}(\pi_{\theta})=-\mathbb{E}_{(x,y_{w})\sim\mathcal{D}}\left[\log\pi_{\theta}(y_{w}\mid x)\right] (2) The combined objective is \mathcal{L}=w\cdot\mathcal{L}_{\text{DPO}}+(1-w)\cdot\mathcal{L}_{\text{SFT}} , where w controls the relative importance of preference learning versus supervised learning. We experiment with different w values in our experiments (cf. Section 3.2 ). 2.3 Constructing SpokenElyza To the best of our knowledge, there are no existing benchmarks to rigorously evaluate the speech-worthiness of Japanese SpeechLLMs. We introduce SpokenElyza , a high-quality evaluation benchmark derived from the standard ELYZA-tasks-100 dataset [ 18 ] . While ELYZA-tasks-100 is widely used for assessing Japanese instruction-following capabilities, it targets text LLMs and contains several examples unsuitable for spoken interaction (e.g., text string manipulation). We constructed SpokenElyza through a four-stage curation pipeline designed to ensure both linguistic naturalness and acoustic suitability: Modality filtering . We screened the original 100 examples to filter out tasks that are inherently non-verbalizable (e.g., text string processing, email/essay writing). We also refined input instructions to reflect natural spoken queries. This filtering process yielded a core set of 36 instruction-response pairs, which we call Elyza (the filtered subset with original written-style responses). Based on Elyza, we then apply speech-worthy style transfer to create SpokenElyza via the next three steps. Speech-Worthy Style Transfer . The original ground-truth responses in Elyza exhibit strong written-text biases, including markdown formatting, bullet points, numbered lists, and excessive length. We employed a strong LLM (gpt-oss-120B) to rewrite responses under strict speech-friendly constraints: (1) removal of all markdown formatting and lists, (2) conversion of written predicates to polite spoken forms, (3) simplification of complex nested sentences, and (4) ensuring comprehensibility solely through audio without visual aids . Two examples required responses that remained excessively long even after rewriting due to the inherent complexity of