MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model

Jingyao Gong · 2026

Read MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model by Jingyao Gong (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-03937.txt.

Introduction

Jingyao Gong reports MiniMind-O, an open small-scale omni model built on the MiniMind language model at about a tenth of a billion parameters. The model accepts text, speech, and image inputs and returns both text and streaming speech. The release includes model code, checkpoints, and the main training datasets, making the complete interaction loop directly inspectable.

Architecturally, a full MiniMind backbone acts as the Thinker, while an independent four-layer Talker built from MiniMind blocks produces speech, reading a middle-layer Thinker state together with an autoregressive codec buffer. Frozen speech and image encoders feed lightweight projectors, and speaker control is handled inside the audio-code context rather than by a separate synthesis module. Dense and mixture-of-experts variants reached similar consistency scores in Thinker-Talker evaluation.

Catalog record

CREATORJingyao Gong
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.03937

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

MiniMind-O Technical Report: An Open Small-Scale Speech-Native Omni Model Jingyao Gong Abstract MiniMind-O is an open 0.1B-scale omni model built on the MiniMind language model ( 8 ; 10 ) . It accepts text, speech, and image inputs, and returns both text and streaming speech. The release includes model code, checkpoints, and the main Parquet training datasets for T2A, I2T, and A2A, making the complete interaction loop directly inspectable. The model uses a full MiniMind backbone as the Thinker and an independent four-layer Talker made from MiniMind blocks. Frozen SenseVoice-Small and SigLIP2 encoders provide speech and image features, which are mapped by lightweight MLP projectors and injected at modality-placeholder positions. The Talker reads a middle-layer Thinker state together with an autoregressive eight-layer Mimi-code buffer. Speaker control is handled by a dedicated speaker token, right-aligned reference codec prompts, and precomputed 192-dimensional CAM++ embeddings, so voice conditioning remains part of the audio-code context rather than a separate TTS module. With a 768-dimensional Talker, the dense and MoE variants reach average CERs of 0.0897 and 0.0900 in Thinker–Talker consistency evaluation, with overall voice-cloning similarities of 0.5995 and 0.5937. Beyond reporting a working system, the paper identifies three scale-critical design choices for small omni models: middle-layer semantic bridging, a released multimodal sequence format, and a parameter-efficient eight-codebook interface. 1 Introduction Models such as GPT-4o, Qwen-Omni, Moshi, and recent speech-text systems have moved real-time multimodal interaction from a product interface problem into a model-design problem ( 17 ; 5 ; 26 ; 27 ) . A usable system has to listen, see, reason, speak, and stop speaking when the user interrupts. The usual engineering path is still a cascade: ASR turns speech into text, an LLM writes the answer, and TTS renders the waveform. This path works, but it leaves the language model outside the acoustic loop. Once the speech module is external, errors in pronunciation, timing, and speaker control are hard to attribute to a shared representation. MiniMind-O takes the opposite constraint as the starting point. The base model is MiniMind, not a billion-scale backbone, so every added modality has to pass through a very small hidden space. This makes the system a useful stress test for omni-model design: components that are merely convenient at large scale have to become explicit and measurable at 0.1B scale. The design in Figure 1 keeps the semantic path and the acoustic path separate. The Thinker is the MiniMind transformer itself. It receives normal text embeddings, plus projected SenseVoice and SigLIP2 states injected at audio and image placeholder positions. The Talker is a separate four-layer module initialized from MiniMind blocks when compatible weights are available. This keeps semantic prediction in the language backbone and gives audio-code generation its own recurrent history. Figure 1: Architecture of MiniMind-O. Audio and image inputs are encoded by frozen SenseVoice and SigLIP2 encoders, mapped into the MiniMind hidden space by MLP projectors, and injected at modality-placeholder positions. A middle-layer Thinker state is fused with the Mimi-code history by an independent Talker, which predicts eight codec layers for streaming speech generation. The size is not only a constraint; it is also the main experimental handle. MiniMind-O is intended as a small and fully inspectable omni implementation: it supports text, speech, and image inputs together with streaming speech output while keeping the active model around 0.1B parameters. At this scale, the bridge, projectors, and codec interface have to remain necessary, measurable, and reproducible. A second result is more architectural. The eight Mimi codebooks could have been given eight independent embedding tables and eight independent output heads. In practice, a non-full-rank shared-base-plus-adapter parameterization gives a clear parameter-efficiency curve: moderate ranks recover most of the convergence and codebook-accuracy gain, while the decoupled rank study shows that the output head rank matters more than the input embedding rank. This makes the low-rank interface an empirically supported design choice rather than an implementation shortcut. The bridge layer is the third point. If the Talker reads the final next-token-prediction state, it inherits a strong bias toward the current text token and the geometry of the LM head; this is useful for text logits but noisy as an acoustic condition. If it reads too shallow a state, the model has not yet accumulated enough context to resolve pronunciation, syntax, or cross-modal reference. A simple Mandarin example is the character U+5730, whose pronunciation can be de or di depending on context. A raw embedding does not encode this context-specific pronunciation, while a middle hidden state can carry enough surrounding information without being fully collapsed into the next-token classifier. The fourth part of the release is the dataset itself. Omni systems are hard to reproduce if the code is open but the alignment data, codec targets, and modality layout are implicit. MiniMind-O therefore releases the main T2A, I2T, and A2A Parquet datasets together with the code path that consumes them. The dataset is not meant to be a final universal corpus; it is the training substrate for this small-model recipe, with text, image bytes, speech inputs, Mimi code targets, reference-code prompts, and speaker embeddings organized in a format that can be inspected and modified. The released system has two variants: a dense minimind-3o model and a minimind-3o-moe model with roughly the same active scale. Audio input is encoded by SenseVoice-Small ( 1 ) ; image input is encoded by SigLIP2 ( 20 ) ; speech output is represented by eight Mimi codebooks and decoded to 24 kHz audio ( 5 ) . Speaker conditioning is injected by two scale-compatible signals: reference codec prompts and 192-dimensional CAM++ speaker embeddings ( 22 ) . This choice also keeps the inference path inspectable, because no speaker encoder is called inside the model forward pass. The voice path is therefore closer to in-context conditioning than to a fixed-speaker TTS head. The default release ships five built-in voice prompts, dylan , eric , serena , uncle_fu , and vivian ; an additional seven voices are kept as held-out prompts for evaluation. At inference time, changing the voice only changes the right-aligned reference Mimi codes and the CAM++ vector placed at the <|audio_spk|> position. The Thinker prompt and Talker weights remain unchanged, which makes voice transfer a property of the shared audio-code layout rather than a separate fine-tuning path. The report documents the design factors identified as important in this small regime: where to extract the Thinker state, how wide the Talker has to be, how reference speech should be placed in the audio buffer, how the released data is organized, and which evaluation exposes content mismatch rather than only audio quality. These details are not incidental implementation choices. At 0.1B scale, bridge placement, reusable data, and parameter-efficient codebook interfaces directly affect whether the complete loop remains trainable and reproducible. The contribution is therefore not a new large model, but a compact and inspectable recipe that turns speech-native omni interaction into a controllable research object. 2 Related Work Figure 2: Talker-side speech generation design. The Talker consumes the Thinker bridge state, audio-code embeddings, optional speaker information, and reference codec prompts, then emits eight-layer Mimi codebook logits for waveform decoding. Omni and speech-text dialogue models. GPT-4o made speech-native multimodal interaction widely visible, and Qwen2.5-Omni and Qwen3-Omni later made the Thinker–Talker

Read the complete text