GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot

Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie Tang · 2024

Read GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot by Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie Tang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-02612.txt.

Introduction

Aohan Zeng leads a team of eight authors in 2024 in presenting GLM-4-Voice, a chatbot that talks and listens in one single model. It speaks Chinese and English, holds real-time voice conversations, and can shift its emotion, tone, speed, and even dialect on request.

At its core is a very compact speech tokenizer that squeezes audio into a thin stream of tokens, built by adding a bottleneck to a speech recognition encoder. To teach the model speech without losing its text skills, the team synthesized training data that weaves speech and text together. The result aims at spoken chat that feels human rather than bolted on.

Catalog record

CREATORAohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie Tang
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-02612

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, Jie Tang Abstract We introduce GLM-4-Voice, an intelligent and human-like end-to-end spoken chatbot. It supports both Chinese and English, engages in real-time voice conversations, and varies vocal nuances such as emotion, intonation, speech rate, and dialect according to user instructions. GLM-4-Voice uses an ultra-low bitrate (175bps), single-codebook speech tokenizer with 12.5Hz frame rate derived from an automatic speech recognition (ASR) model by incorporating a vector-quantized bottleneck into the encoder. To efficiently transfer knowledge from text to speech modalities, we synthesize speech-text interleaved data from existing text pre-training corpora using a text-to-token model. We continue pre-training from the pre-trained text language model GLM-4-9B with a combination of unsupervised speech data, interleaved speech-text data, and supervised speech-text data, scaling up to 1 trillion tokens, achieving state-of-the-art performance in both speech language modeling and spoken question answering. We then fine-tune the pre-trained model with high-quality conversational speech data, achieving superior performance compared to existing baselines in both conversational ability and speech quality. The open models can be accessed through https://github.com/THUDM/GLM-4-Voice and https://huggingface.co/THUDM/glm-4-voice-9b . 1 1 footnotetext: Equal contribution. Email: {zah22,zx-du20}@mails.tsinghua.edu.cn 4 4 footnotetext: Work was done when ML, LZ interned at Zhipu.AI. 1 Introduction The success of large language models (LLMs) has driven significant advancements in conversational AI, enabling the development of text-based chatbots and digital assistants. However, LLMs are primarily designed to process text input and generate text output, focusing on semantic and logical communication. In contrast, human communication extends beyond semantics, often conveying emotions and subtle nuances. Voice-based interaction, therefore, provides a more natural and intuitive medium for human-computer interaction, offering richer and more engaging user experiences. Traditional spoken chatbot typically rely on a pipeline combining Automatic Speech Recognition (ASR), LLM processing, and Text-to-Speech (TTS) synthesis. While functional, this approach is often hindered by high latency, compounded errors introduced during the ASR and TTS stages, and a limited capacity to capture and express emotional nuances. Speech-language models (SpeechLMs), which process both speech input and output in an end-to-end manner, offer a promising approach for building spoken chatbots. Efforts such as [ 24 , 17 ] have explored pre-training on speech data in a manner similar to large language models (LLMs). Similarly, Défossez et al. 2024 scaled speech data to 7 million hours for model training. However, these approaches face a significant limitation: the relative scarcity of speech data compared to the extensive text corpora available online. This data imbalance makes it challenging to fully leverage the capabilities of text-based LLMs, ultimately constraining the intelligence of SpeechLMs. Other methods aim to align speech and text modalities [ 15 , 42 ] by integrating a speech encoder and a text-to-speech module into existing LLMs and fine-tuning them on spoken dialogue datasets. While this approach provides a straightforward way to develop speech-to-speech models from LLMs, it lacks the ability to deliver truly human-like speech output due to the absence of dedicated speech pre-training. This limitation hinders these models from capturing the rich nuances and expressiveness inherent in human speech. In this paper, we introduce GLM-4-Voice, an intelligent and human-like spoken chatbot. We use a single code-book supervised speech tokenizer with 12.5Hz frame rate to efficiently represent speech. A flow-matching-based speech decoder is employed to convert speech tokens into natural-sounding speech. To bridge the gap between text and speech modalities, we conduct large-scale speech-text pre-training using 1 trillion tokens. This includes synthetic interleaved speech-text corpora derived from text pre-training data, as well as unsupervised speech data and supervised speech-text datasets (e.g., ASR and TTS). The resulting base model demonstrates strong performance across various tasks, including speech language modeling, spoken question answering, ASR, and TTS. To further enhance the chatbot’s conversational capabilities, we fine-tune the base model on high-quality conversational datasets using a "streaming thoughts" template. This template alternates between outputting text and speech tokens, improving the model’s ability to generate seamless, low-latency responses while maintaining high-quality performance. 2 Related Work 2.1 Speech Tokenization Speech tokenizers, which transform a audio clip into discrete tokens, can be categorized into two directions. The neural acoustic codecs [ 44 , 11 , 23 , 20 ] target at reconstructing high-quality audio at low bitrates. The semantic tokens [ 19 , 10 ] are extracted from speech representations learned with self-supervised learning on speech data. Recently, SpeechTokenizer [ 48 ] and Mini [ 12 ] unify semantic and acoustic tokens as different residual vector quantization (RVQ) layers, but they also suffer from multiple tokens at the same position, leading to either parallel prediction of semantic and acoustic tokens, or degradation to semantic tokenizers for language models. CosyVoice [ 14 ] proposes the supervised semantic tokenizer derived from a speech recognition model, and successfully apply the tokenizer to text-to-speech synthesis. The application of the tokenizer on speech language modeling is not explored. 2.2 Speech Language Modeling Speech language models are autoregressive models pretrained on unsupervised speech data. Lakhotia et al. 2021 first proposes generative spoken language modeling (GSLM), which trains the next-token-prediction objective on discrete semantic tokens produced by self-supervised learning. AudioLM [ 5 ] proposes a hybrid tokenization scheme that combines these semantic tokens with acoustic tokens from a neural audio codec [ 44 ] . TWIST [ 17 ] trains the speech language model using a warm-start from the pretrained text language model OPT [ 47 ] . Moshi [ 12 ] scales up the size of natural speech data in TWIST to 7 million hours. Spirit-LM [ 32 ] further extends TWIST by adding speech-text interleaving data curated from speech-text parallel corpus. However, the scarcity of speech-text parallel corpus restricts the scale of interleaving data. 2.3 End-to-End Spoken Chatbots Early works in speech-to-speech models mainly focus on processing tasks like speech translation [ 8 , 2 ] . Since success of ChatGPT in text-based chatbots, many works have explored methods to develop speech-based chatbots that can understand and respond in speech. SpeechGPT [ 46 ] proposes to combine existing large language models (LLM) with discrete speech representations to obtain speech conversational abilities. Moshi [ 12 ] proposes a full-duplex spoken dialogue framework based on their pretrained speech language model. Qwen-Audio [ 9 ] adapts pre-trained textual language models for speech understanding by aligning speech representations of the Whisper [ 36 ] encoder. The model can understand speech, but not generate speech. Llama-Omni [ 15 ] and Freeze-Omni [ 41 ] extend the method by adding a text-to-speech model after the language model to transform the text output to speech output. In this way language models can only control the content of speech, but not the styles and prosodies. Mini-Omni [ 42 ] directly fine-tunes language models to generate text and speech responses simultaneously with only instruction datasets. Without speech pre-training, the quality of both text and speech responses is severely limited, as we will show in the experiments. 3 Architecture In this section, we introduce the architecture of GLM-4-Voice. Our goal is to build a human-like, end-to-end spoken chatbot with high intelligence. To achieve this, the model must 1) comprehend the user’s speech and provide a semantically accurate response, and 2) follow the user’s spoken instructions,

Read the complete text