Human-1 by Josh Talks : A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations

Bhaskar Singh, Shobhit Banga, Pranav Sharma · 2026

Read Human-1 by Josh Talks : A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations by Bhaskar Singh, Shobhit Banga, Pranav Sharma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-23295.txt.

Introduction

Singh and three colleagues build Human-1, the earliest open, reproducible full-duplex spoken dialogue system for Hindi, adapting the Moshi architecture. A custom Hindi tokeniser and 26,000 hours of real conversations from 14,695 speakers with separate channels teach turn-taking and overlap directly. Replacing the English tokeniser while keeping pretrained audio parts, they train in two stages: large-scale pretraining, then tuning on 1,000 hours of chat data. Automatic and human tests through prompted dialogue continuation show natural, meaningful full-duplex Hindi chat.

Catalog record

CREATORBhaskar Singh, Shobhit Banga, Pranav Sharma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18578
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.23295

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Human-1 by Josh Talks : A Full-Duplex Conversational Modeling Framework in Hindi using Real-World Conversations Bhaskar Singh, Shobhit Banga, Pranav Sharma Abstract Full-duplex spoken dialogue systems can model natural conversational behaviours such as interruptions, overlaps, and backchannels, yet such systems remain largely unexplored for Indian languages. We present the first open, reproducible full-duplex spoken dialogue system for Hindi by adapting Moshi, a state-of-the-art duplex speech architecture, using a custom Hindi tokeniser and training on 26,000 hours of real spontaneous conversations collected from 14,695 speakers with separate speaker channels, enabling direct learning of turn-taking and overlap patterns from natural interactions. To support Hindi text generation, we replace the original English tokeniser and reinitialise text-vocabulary-dependent parameters while retaining the pre-trained audio components. We propose a two-stage training recipe—large-scale pre-training followed by fine-tuning on 1,000 hours of conversational data. Evaluation through the prompted dialogue continuation paradigm with both automatic metrics and human judgements demonstrates that the resulting model generates natural and meaningful full-duplex conversational behaviour in Hindi. This work serves as a first step toward real-time duplex spoken dialogue systems for Hindi and other Indian languages. Index Terms : spoken dialogue system, full-duplex, speech-to-speech, language adaptation 1 Introduction Human conversation is inherently full-duplex: speakers routinely overlap, produce backchannels, and interrupt one another in ways that half-duplex systems—which wait for one speaker to finish before responding—cannot capture [ 1 , 2 , 3 ] . Recent end-to-end speech dialogue models have begun to address this limitation by jointly modeling both speaker streams in parallel [ 4 , 5 , 6 , 7 ] . However, progress in full-duplex spoken dialogue has been concentrated almost exclusively on English, with limited exploration in other languages. This leaves a significant gap for languages with large speaker populations, distinct phonological systems, and different conversational norms. Hindi, spoken by over 600 million people, is one such language. Hindi conversations are characterised by frequent backchannels, code-mixing with English, and prosodic patterns that differ substantially from those found in English dialogue [ 8 , 9 ] . Building a full-duplex dialogue system for Hindi presents several technical challenges. First, existing duplex architectures use tokenisers trained on English (or other Latin-script languages), which are inefficient for Devanagari script and produce excessive token fragmentation. Second, replacing the tokeniser necessitates reinitialisation of vocabulary-dependent model parameters, fundamentally changing the training dynamics from standard fine-tuning to partial re-training, as these parameters must be learned from scratch while the remaining pre-trained weights are preserved. Third, and most critically, training full-duplex models requires large-scale stereo recordings with separate speaker channels capturing natural turn-taking, overlaps, and backchannels. The quality and scale of such data is arguably the most important factor in the performance of duplex dialogue systems, yet such large-scale stereo conversational resources remain largely unavailable for Hindi and most Indian languages. In this paper, we address these challenges. We collect a large-scale corpus of 26,000 hours of real Hindi spontaneous conversations from 14,695 speakers with separate channel recordings—to our knowledge, the largest purpose-built conversational speech corpus for any Indian language—and present a framework for adapting full-duplex speech dialogue architectures to Hindi. Our results demonstrate that Real world natural spontaneous conversational data is the critical enabler for building effective full-duplex dialogue systems in new languages. 2 Related Work The dGSLM model [ 1 ] was among the first to model two-channel spoken dialogue jointly. Moshi [ 4 ] advanced this by combining a 7B-parameter text LLM with a neural audio codec (Mimi) and a hierarchical RQ-Transformer to achieve real-time full-duplex dialogue in English. Other recent efforts include SyncLLM [ 5 ] and OmniFlatten [ 6 ] . These systems have been developed primarily in English. Cross-lingual adaptation has been explored for ASR [ 12 ] and TTS [ 13 ] , but adapting full-duplex dialogue models to new languages remains unexplored. SpeechGPT [ 10 ] and AudioPaLM [ 11 ] demonstrated multilingual speech capabilities but in a turn-based setting. Neural audio codecs such as Mimi are trained predominantly on English; whether they generalise to typologically different languages is an open question. To our knowledge, no prior work has presented an open full-duplex spoken dialogue system for any Indian language. 3 Method 3.1 Base Architecture: Moshi We build upon Moshi [ 4 ] , a full-duplex spoken dialogue model with three components. Mimi is a neural audio codec that encodes 24 kHz speech into discrete tokens at 12.5 Hz using 8 codebook layers—layer 1 captures semantic content while layers 2–8 capture acoustic detail. The RQ-Transformer is a hierarchical architecture: the Temporal Transformer (7B parameters, based on Helium) models 17 parallel streams per timestep (1 text + 8 Moshi audio + 8 user audio), producing a hidden vector z_{s} from which a text token is sampled via the Text Linear layer; the Depth Transformer then autoregressively generates 16 audio tokens conditioned on z_{s} . PAD tokens are inserted at timesteps without corresponding text, and a one-step acoustic delay stabilises generation. Preliminary evaluation confirms that Mimi achieves acceptable reconstruction on Hindi without retraining (Section 5.1 ), so we freeze Mimi throughout and train only the RQ-Transformer, substantially reducing computational cost. 3.2 Hindi Tokeniser and Parameter Reinitialisation The original architecture uses a SentencePiece tokeniser [ 16 ] trained on English text with a 32,000-word vocabulary. This tokeniser is fundamentally incompatible with Hindi’s Devanagari script, producing excessive fragmentation that degrades both training efficiency and language modelling quality. We replace it with a SentencePiece model trained on a large Hindi text corpus, also with a 32,000-word vocabulary. This replacement requires reinitialisation of all vocabulary-dependent parameters which includes Text token embedding tables in both the Temporal and Depth Transformers. And The Text Linear projection layer that maps hidden states to token probabilities. This reinitialisation has a critical consequence: while audio processing components retain their pre-trained weights, all text representations must be learned from scratch. The task is therefore partial pre-training rather than fine-tuning, which requires substantially different hyperparameter choices (Section 3.4 ). 3.3 Data 3.3.1 Pre-training Data A key contribution of this work is the collection of a large-scale Hindi conversational speech corpus specifically designed for full-duplex dialogue modelling. We collected 26,000 hours of real two-person Hindi spontaneous conversations involving 14,695 unique speakers. The data was gathered through a dedicated collection effort with hired participants engaging in spontaneous, unscripted conversations on diverse topics. This corpus forms part of our broader, ongoing infrastructure program to collect 1 million hours of natural, spontaneous conversational speech in Hindi - building the scale of data required to support next-generation conversational and full-duplex speech systems. Each conversation was recorded with separate speaker channels, providing natural stereo dialogue without the need for artificial speaker diarisation—a significant advantage over prior approaches that construct pseudo-stereo data from monophonic recordings [ 4 ] . Domain-matched data. A critical design decision was collecting data in a conversational setting rather than compiling read speech, broadcast news, or monologue recordings. Full-duplex dialogue models must learn turn-taking, backchannels, interruptions, and overlaps—phenomena that only occur naturally in real conversations. Our corpus directly captures these dynamics in its native stereo format, eliminating the noise and artifacts introduced by post-hoc diarisation of monophonic sources. Quality assurance. The collected data underwent extensive quality checks by trained annotators who evaluated recordings against predefined criteria

Read the complete text