OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder
Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei · 2026
Read OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder by Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-01506.txt.
Introduction
Detao Bai and six co-authors build one encoder that takes in sight and sound together, the way people do. Current omni-modal models use separate encoders and a lopsided diet. Video is sampled at a frame or two per second against audio many times denser, so motion detail and cross-sense timing are lost.
Their OmniEncoder runs a single Transformer backbone that embeds both streams at a matched 25 frames per second in one shared space. Three devices make it work: a token template that tags what is what, a position scheme tuned for mixed streams, and a shifting time window that keeps cost sane. Against a strong split-encoder baseline on the same token budget, the unified model gains a lot on smooth visual tasks. Those tasks include sign language and fine sports actions, and the model holds its own on audio-visual tests.
Catalog record
| CREATOR | Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.01506
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei Abstract Recent advances in omni-modal large language models have enabled remarkable progress in joint vision-audio understanding. However, prevailing architectures rely on modality-specific encoders with a video-coarse, audio-dense design—sampling visual frames at 1–2 fps while processing audio waveforms at 25 fps—resulting in systems that perceive video frame by frame, modality by modality rather than holistically as humans do. Such a discrepancy leaves models with impoverished cross-modal interaction during encoding and an inability to capture fine-grained visual motion. To bridge this gap, we present Omni-Encoder, a unified Transformer backbone designed to co-embed visual and audio signals at a symmetrical 25 fps within a shared latent space. This architecture leverages three core innovations—the Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting—to effectively reconcile the dual challenges of modality disentanglement and computational efficiency. Experiments demonstrate that, compared to the modality-specific baseline Qwen2.5-Omni under the same input token budget to the LLM decoder, Omni-Encoder delivers substantial gains on visual continuous understanding tasks—such as sign language recognition and fine-grained sports action analysis—while maintaining competitive performance on established audio-visual benchmarks such as AVQA and Speaker Identification and Localization. These results suggest that unified omnivorous encoding offers a promising direction for building omni-modal models that more closely reflect the integrated nature of human perception. 1 Introduction Figure 1: (a). Modality-specific encoders: process visual and audio information separately at mismatched frame rates (1–2 fps for video, 25 fps for audio). (b). Omni-Encoder: a single Transformer jointly encodes audio, visual base, and visual continuous tokens at 25 fps within a unified representation space. (c). With the same number of input tokens to the LLM, we compare Qwen2.5-Omni-3B with the original encoder and Omni-Encoder, both trained with SFT, on tasks requiring visual continuous understanding (e.g. sign language, sports action) and audio-visual QA tasks. Human-beings understand the world is inherently omnimodal, involving the integration of visual, auditory, and linguistic information. Benefiting from the rapid advances in vision [ 1 , 2 , 3 ] , audio [ 4 , 5 , 6 ] , and language [ 7 , 8 ] models, omni-modal LLMs [ 9 , 10 , 11 , 12 , 13 , 14 , 15 , 16 ] have grown remarkably capable at vision-audio understanding. Most OmniLLMs typically follow a modality-specific modular design. Specifically, a visual encoder [ 2 , 3 ] , an audio encoder [ 4 , 6 ] , and modality-specific projectors are utilized to encode each modality’s information separately (Fig. 1 a). While this design philosophy effectively leverages modality-specific pretraining, such technological inertia results in omni-models that perceive video “frame by frame, modality by modality”. Yet this does not reflect how humans perceive video. In contrast, humans see, hear, and feel continuous motion simultaneously. Cross-modal interactions occur at the earliest stages of sensory processing in the central nervous system [ 17 , 18 ] . These observations naturally motivate a fundamental question: Is it possible to jointly encode visual and audio information within a unified omnivorous encoder? Achieving this presents two key challenges. First, processing multiple modalities within a unified encoder requires the model to distinguish modality-specific representations, support diverse encoding modes (visual-only, audio-only, and visual-audio), and retain the strong extrapolation capabilities of prior modality-specific encoders (e.g., support for arbitrary resolutions and temporal lengths). Second, constrained by the computational cost of bidirectional attention, existing OmniLLM works adopt a video-coarse, audio-dense design — for instance, in Gemini and Qwen2.5-Omni, video is sampled at 1–2 fps while audio is processed at 25 fps. For videos with dense motion dynamics (e.g., sign language recognition, fine-grained sports such as gymnastics, or lip reading),it is desirable for the model to capture fine-grained motion at a frame rate comparable to audio, while maintaining acceptable computational overhead. To this end, we introduce Omni-Encoder (Fig. 1 b), a single Transformer backbone that jointly encodes visual frames and audio waveforms within one representation space at high frame rate (25fps). The key designs of our Omni-Encoder are threefold: • Omni-Encoder Token Template. We propose a unified omni-modal encoding template that decomposes raw 25 fps video into three token streams—Audio, Visual Continuous (VC), and Visual Base (VB) tokens—at each temporal position. VC tokens are introduced as frame-wise learnable queries that capture inter-frame motion dynamics, such as motion trajectories, gesture onsets, and micro-movements. The encoder output passes through a Token Sparsifier that reduces VB tokens to 2 fps while preserving Audio and VC tokens at full 25 fps, drastically cutting the token count forwarded to the downstream decoder without sacrificing motion fidelity. • Omni-RoPE. A 3D rotary positional encoding that assigns each token a unique coordinate (t,h^{\prime},w^{\prime}) in a unified spatio-temporal-modality space. This formulation enables the Omni-Encoder to distinguish heterogeneous Audio, Visual Continuous, and Visual Base tokens and model their cross-modal relationships, while preserving resolution-agnostic extrapolation for arbitrary visual resolutions and temporal lengths. • Temporal Window Shifting. We propose a joint spatiotemporal attention mechanism that alternates between local and shifted temporal windows across layers, reducing complexity from \mathcal{O}(T^{2}) to \mathcal{O}(T\cdot G) . This enables dense cross-modal interaction at 25 fps while keeping computation linear in video length. 2 Architecture As illustrated in Figure 2 , Omni-Encoder is a 24-layer Transformer backbone that natively processes visual frames, audio waveforms, and continuous motion signals at the original video frame rate of 25 fps. Rather than encoding each modality in isolation and fusing them at a later stage, our approach co-embeds all modalities into a unified sequence and applies joint self-attention from the first encoder layer, allowing cross-modal alignment to emerge during encoding rather than as a downstream patch. This design mirrors how humans integrate sight, sound, and motion to interpret social interactions—enabling holistic, temporal perception from raw video input. Figure 2: Architecture of Omni-Encoder. A 24-layer Transformer jointly encodes Audio, Visual Continuous, and Visual Base tokens from raw 25 fps video through unified self-attention, with each layer incorporating Omni-RoPE and Temporal Window Shifting. A Token Sparsifier then sparsifies Visual Base tokens to match the native input length of Qwen2.5-Omni. The encoded token sequence passes through a Token Sparsifier before entering the Qwen-LLM 3B decoder for downstream reasoning. Critically, the Token Sparsifier strategically retains all Audio and Visual Continuous tokens at full temporal resolution while sparsely sampling Visual Base tokens, reducing the total token count to match the native input length of Qwen2.5-Omni [ 11 ] . This ensures full compatibility with the pretrained decoder without any architectural modification, while preserving high temporal resolution for capturing fine-grained behavioral cues. To enable efficient processing of native 25 fps video, Omni-Encoder incorporates three key architectural designs: Omni-Encoder Token Template, Omni-RoPE, and Temporal Window Shifting. We describe each component in the following subsections. 2.1 Omni-Encoder Token Template:Jointly Encoded Audio, VC & VB Tokens As shown in Figure 2 , input video is decomposed into three distinct token streams: • Audio Tokens( \mathbf{a}_{t} ): Audio tokens are extracted from audio log-Mel spectrograms using two lightweight 1D CNNs, producing discrete audio embeddings aligned with video frames. • Visual Continuous Tokens( \mathbf{v}_{t}^{\mathrm{c}} ): VC tokens are introduced as frame-wise learnable vectors at each temporal position, capturing inter-frame information during encoding