JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, Fan Zhou, Liang Zheng, Shuicheng Yan, Hao Fei, Tat-Seng Chua · 2025
Read JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation by Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, Fan Zhou, Liang Zheng, Shuicheng Yan, Hao Fei, Tat-Seng Chua (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-22905.txt.
Introduction
Kai Liu and a large team introduce JavisGPT, a multimodal language model that both understands and creates videos with sound. Earlier systems handled silent video or treated audio as an afterthought, so keeping speech, effects, and picture in step was hard. JavisGPT links an encoder, a large language model, and a pretrained audio-video diffusion generator through a fusion module designed around synchronization.
Training proceeds in three stages, moving from general multimodal pretraining through audio-video fine-tuning to large-scale instruction tuning. For the last stage the team built JavisInst-Omni, a dataset of more than two hundred thousand dialogues about audio-video content, curated with GPT-4o. On benchmarks for joint audio-video understanding and generation, JavisGPT outperformed existing multimodal models, with the clearest gains in complex scenes where sound and image must stay tightly synchronized over time.
Catalog record
| CREATOR | Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, Fan Zhou, Liang Zheng, Shuicheng Yan, Hao Fei, Tat-Seng Chua |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 80 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-22905
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation Kai Liu, Jungang Li, Yuchong Sun, Shengqiong Wu, Jianzhang Gao, Daoan Zhang, Wei Zhang, Sheng Jin, Sicheng Yu, Geng Zhan, Jiayi Ji, Fan Zhou, Liang Zheng, Shuicheng Yan, Hao Fei, Tat-Seng Chua Abstract This paper presents JavisGPT , the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni , a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings. Project: https://JavisVerse.github.io/JavisGPT-page Figure 1 : JavisGPT supports multi-level synchronized audio-video (SyncVA) content ( i.e. , sounding video) comprehension ( left ), and simultaneously complex instruction-based SyncVA generation, interleaved in-context generation ( middle ), and multi-turn proactive conversations ( right ). 1 Introduction Joint understanding and generation of multiple modalities has recently sparked increasing interest [ 72 , 79 , 21 , 71 , 86 , 50 ] . Existing efforts generally focus on image-text frameworks [ 96 , 84 , 73 , 51 , 11 ] , or treat video and audio understanding and generation as separate modalities [ 90 , 80 , 42 ] . Figure 2 : Overall architecture of JavisGPT , which can perceive and produce videos and sounds simultaneously. The input video and audio are tokenized and fed into the SyncFusion module. The resulting synchronized audio-video representation, together with the text tokens and learnable JavisQueries are then passed to the LLM backbone. During decoding, the yielded JavisCond embeddings are used to align the LLM intents to the semantic conditional space of downstream JAV-DiT, enabling high-quality and synchronized sounding-video generation. We introduce JavisGPT , a bespoke MLLM for high-quality, unified comprehension and generation in joint audio-video (JAV) tasks, which has clear application potential in avatar chatbots, sounding movie curation, and music-driven video analysis. JavisGPT is a holistic system that accepts as input separate audio and videos, synchronized sounding videos, and user text prompts. After comprehending and reasoning over these input signals, it produces either textual responses and/or synchronized sounding videos. Our architecture supports a wide range of JAV scenarios, including question-answering in comprehending sounding audios, generation of sounding audios following user instructions, upon in-context reasoning, and interleaved text-audio-video generation in multi-turn user dialogues (ref. Figure 1 ). As a comprehensive and state-of-the-art system, we highlight the design of JavisGPT below (with detailed discussions with related works provided in Section 2 ). Model architecture. JavisGPT uses an encoder-LLM-decoder architecture (ref. Figure 2 ), with Qwen2.5 [ 88 ] as the LLM backbone. The visual encoder is inherited from Qwen2.5-VL [ 5 ] , and the audio encoder is based on BEATs [ 10 ] . Audio and video features, along with user prompts and learnable JavisQuery tokens, are passed to the LLM. To enable fine-grained spatiotemporal alignment, we propose a dedicated SyncFusion module that fuses audio and video representations into synchronized SyncAV tokens for unified comprehension. At the output stage, the LLM generates textual responses along with JavisCond tokens, which encode contextual semantics and serve as conditioning inputs for a pretrained JAV-DiT generator [ 39 ] . We choose JavisDiT for its generation quality and flexibility, and incorporate hierarchical JavisQueries to provide spatiotemporal priors, further enhancing synchronization in audio-video generation. Training strategies. To effectively adapt the vision-language backbone to JAV tasks, we design a three-stage training pipeline: (1) MM-PreTrain : introducing the audio input branch for comprehension and preliminarily aligning the output embeddings of LLM with the condition space of JavisDiT; (2) AV-FineTune : enhancing synchronized audio-video comprehension and generation via SyncFusion and hierarchical JavisQueries, respectively; and (3) MM-InstTune : enables generalizable instruction following and multimodal reasoning via large-scale instruction tuning across text, video, and audio. Dataset collection. To facilitate instruction tuning, we construct JavisInst-Omni , a large-scale, diverse, and high-quality instruction dataset covering both comprehension and generation. The dataset contains 200K dialogue trajectories involving tightly interleaved text, audio, and video, simulating complex single- and multi-turn interactions. All samples are annotated via GPT-4o and manually verified for quality. The instructions span a wide range of task formats, including QA, captioning, instruction-following generation, and in-context multimodal reasoning. We compare JavisGPT with existing JAV-like LLMs on various JAV benchmarks, where JavisGPT yields impressive performance across all scenarios. Moreover, for audio-video joint generation, JavisGPT also significantly outperforms existing methods, producing high-quality and diverse sounding videos. We reiterate our main points below: • We present JavisGPT, the first unified MLLM for sounding-video comprehension and generation. • We present its architecture design, including the audio-video synchronization mechanism, its multi-stage tuning strategy, and a large-scale instruction dataset for JAV LLMs training. • Evaluation indicates the effectiveness of JavisGPT in JAV comprehension and generation. 2 Related Work Multimodal comprehension. Recently, MLLMs have significantly advanced many multimodal fields, such as the modeling of image, video, and audio contents [ 5 , 12 , 17 , 60 , 22 , 20 , 81 , 55 ] . Early approaches, such as Flamingo [ 4 ] and BLIP [ 34 , 33 ] , focused on combining text with static visual inputs. Recent research extended these efforts to dynamic vision and audio modalities, such as PandaGPT [ 67 ] , Video-LLaMA [ 91 , 14 ] , UnifiedMLLM [ 36 ] , and MiniOmni [ 85 ] , which incorporate temporal and auditory perception and reasoning. While these MLLMs achieve promising performance, they typically treat modalities ( e.g. , audio and video) as independent inputs from separate channels, relying on concatenation or simple feature padding for integration. This strategy overlooks fine-grained spatio-temporal interactions between modalities, compromising their effectiveness in complex scenarios where multimodal synchronization is required, such as JAV. Multimodal generation. Compared to research efforts for multimodal content comprehension, relatively limited emphasis has been placed on generative capabilities for MLLMs, particularly for scenarios involving multiple interdependent modalities, such as JAV [ 31 , 55 ] . For audio-video generation, existing MLLMs like HuggingGPT [ 64 ] , NExT-GPT [ 80 ] , and UnifiedIO-2 [ 42 ] adopt a pipeline-based framework, where individual audio and video generation components are sequentially executed. Although such methods achieve reasonable results, they suffer from issues like error propagation and desynchronized outputs due to the lack of joint modeling. Unified comprehension and generation. Early approaches such as AV2AV [ 15 ] and MultiDialog [ 52 ] tackled talking-head generation and speech-driven dialogue, but were limited by simple architectures or the absence of textual input/output. While recent MLLMs aim to unify multimodal comprehension and generation, most still focus on single modalities, lacking effective modeling of cross-modal interactions, particularly the audio-video synchronization. For instance, Video-Salmonn [ 68 , 69 ] and Meerkat [ 16 ] use separate encoders for audio and