NEXUS-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision

Che Liu ∗ \text{Che Liu}^{*}, Yingji Zhang ∗ \text{Yingji Zhang}^{*}, Dong Zhang* Weijie Zhang* Chenggong Gong*, Yu Lu, Shilin Zhou, Ziliang Gan, Ziao Wang, Haipang Wu Ji Liu, Andre Freitas, Qifan Wang, Zenglin Xu, Rongjunchen Zhang † \dagger, Yong Dai † ‡ \text{Yong Dai}^{\dagger}\ddagger · 2025

Read NEXUS-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision by Che Liu ∗ \text{Che Liu}^{*}, Yingji Zhang ∗ \text{Yingji Zhang}^{*}, Dong Zhang* Weijie Zhang* Chenggong Gong*, Yu Lu, Shilin Zhou, Ziliang Gan, Ziao Wang, Haipang Wu Ji Liu, Andre Freitas, Qifan Wang, Zenglin Xu, Rongjunchen Zhang † \dagger, Yong Dai † ‡ \text{Yong Dai}^{\dagger}\ddagger (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01879.txt.

Description

One model that can see, hear, and talk at once is an industry goal, and Che Liu and co-authors present their entry in a 2025 paper. Their pipeline lets builders mix and match encoders, a language core, and decoders like interchangeable parts. Instead of training vision from scratch, they bolt ears onto a strong existing vision-language model.

A synthetic speech factory supplies extra training audio drawn from everyday scenes. The resulting model beat its own vision backbone at image understanding, led a spoken question benchmark among its peers, and held its own at speech making. Adding ears, they found, even tightened how sight and language line up inside.

Catalog record

CREATORChe Liu ∗ \text{Che Liu}^{*}, Yingji Zhang ∗ \text{Yingji Zhang}^{*}, Dong Zhang* Weijie Zhang* Chenggong Gong*, Yu Lu, Shilin Zhou, Ziliang Gan, Ziao Wang, Haipang Wu Ji Liu, Andre Freitas, Qifan Wang, Zenglin Xu, Rongjunchen Zhang † \dagger, Yong Dai † ‡ \text{Yong Dai}^{\dagger}\ddagger
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01879

Browse the full library · About the Tower of Babel