OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL
Karl El Hajal, Mathew Magimai.-Doss · 2026
Read OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL by Karl El Hajal, Mathew Magimai.-Doss (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-30356.txt.
Introduction
Karl El Hajal and Mathew Magimai-Doss teach speech models to both analyze and synthesize, arguing the two skills reinforce each other. Self-supervised speech models usually train for recognition-style tasks alone. Signal-level detail needed for speaking is not kept. Their framework, OLIVE, joins masked latent prediction with waveform rebuilding. View changes supply the invariance pressure during training.
Rebuilding the waveform forces early encoder features to keep signal-level information. That prediction shapes later representations toward robustness. The unified objective balances analysis and synthesis in one model. Results improve most on generation and speaker related tasks. Recognition and semantic task performance stays competitive with specialists. Waveform quality itself also improves under OLIVE. The authors frame this as reviving the classical analysis-synthesis paradigm. Those methods give that old pairing new power.
Catalog record
| CREATOR | Karl El Hajal, Mathew Magimai.-Doss |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.30356
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
OLIVE: View-Augmented Latent Prediction with Waveform Reconstruction for Speech SSL Karl El Hajal, Mathew Magimai.-Doss Abstract We propose O nline L atent prediction with I nvariant V iews and r E construction (OLIVE), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked latent prediction with waveform reconstruction under a unified objective. Reconstruction constrains early encoder features to retain signal-level information, while masked latent prediction shapes later contextual representations toward invariance for robust downstream performance. We show that these objectives enable representations that support a broad range of tasks. In particular, OLIVE improves results on generation and speaker tasks, maintains competitive performance on recognition and semantic tasks, and improves waveform reconstruction. 1 Introduction Recent advances in speech representation learning have been largely driven by self-supervised learning (SSL) on large-scale unlabeled audio [ 39 , 37 ] . Models such as wav2vec 2.0 [ 8 ] , HuBERT [ 26 ] , WavLM [ 12 ] , and data2vec [ 6 , 5 ] learn contextual representations that serve as general-purpose feature extractors, supporting both linguistic tasks such as automatic speech recognition [ 8 , 26 ] and paralinguistic tasks including speaker and other non-linguistic acoustic attributes such as diarization and speech emotion recognition [ 61 , 9 , 58 , 3 , 44 ] . These methods inherit ideas from contrastive prediction [ 54 , 48 ] , masked modeling [ 17 , 32 , 36 ] , quantization and discrete target discovery [ 32 , 14 , 33 ] , and teacher–student distillation [ 23 , 40 , 10 ] . Existing approaches are nevertheless predominantly focused on discriminative tasks: they aim to extract representations that are maximally informative for downstream predictive tasks, without explicitly preserving the information required for speech signal generation. In contrast, speech communication fundamentally arises from two tightly coupled phenomena: perception (hearing) and production (speaking). Classical speech processing has long embraced this duality through the analysis–synthesis paradigm, where signals are decomposed into structured components and subsequently reconstructed [ 2 , 38 , 22 ] . This framework leverages the fact that speech production is governed by a well-defined physical and physiological process, making it amenable to explicit modeling [ 20 , 51 , 65 ] . Recent neural work has revisited this perspective by training decoders on frozen self-supervised representations, effectively forming a neural analysis–synthesis pipeline [ 13 , 42 , 49 , 24 ] . This has enabled a range of synthesis approaches for tasks such as voice conversion [ 4 ] , text-to-speech [ 18 ] , and speech enhancement [ 28 ] that leverage the structured information encoded in intermediate self-supervised representations. Despite this motivation, synthesis remains peripheral in most modern speech SSL systems. Recent efforts have begun to incorporate generative objectives, for example through auxiliary reconstruction losses or jointly trained encoder–decoder models [ 57 , 34 ] . These developments suggest that analysis and synthesis can be coupled during pre-training, but leave open how to model waveform-to-waveform reconstruction directly, while retaining a functional decoder within the pre-trained model, and preserving the discriminative strength of speech SSL. In this work, we propose OLIVE ( O nline L atent prediction with I nvariant V iews and r E construction), a self-supervised speech representation learning framework that jointly optimizes analysis and synthesis objectives. OLIVE combines view-augmented masked prediction in the latent space with waveform reconstruction under a unified objective. OLIVE therefore explicitly balances two training goals: the synthesis branch constrains early encoder features to preserve information for waveform reconstruction, while the analysis branch shapes later contextual representations toward invariance for robust downstream analysis. This stands in contrast to purely discriminative models, which may discard information critical for generation, and purely generative approaches, which may underperform on discriminative tasks. We validate our method against established SSL baselines and demonstrate that jointly modeling analysis and synthesis yields competitive downstream performance while enabling high-quality waveform reconstruction. Code and model weights will be made publicly available upon publication. Contributions. • We introduce OLIVE , a speech SSL framework that jointly optimizes analysis and synthesis in a single pre-training stage. • We propose a design in which waveform reconstruction acts on early encoder features, while later contextual representations are shaped primarily by view-augmented masked prediction in the latent space. • We demonstrate that this design improves waveform reconstruction and downstream performance on generative and speaker-related tasks while maintaining competitive results on recognition and semantic tasks. 2 Related Work Self-supervised speech representation learning. Early speech SSL methods learned from raw waveforms using contrastive objectives that predict latent representations of future or masked inputs from past context [ 54 , 48 , 55 , 7 , 15 ] . Wav2vec 2.0 introduced masked latent prediction with quantized targets and a contrastive loss at masked time steps [ 8 ] . HuBERT replaced online quantization with masked prediction of offline clustered units [ 26 ] , while WavLM extended this line with denoising and large-scale pre-training for a broader set of speech tasks [ 12 ] . Data2vec instead predicts continuous contextualized teacher representations, avoiding discrete targets [ 6 ] . Data2vec 2.0 improves the efficiency of this masked self-distillation setup by reusing teacher targets across masked views, encoding only unmasked steps, and using a lightweight convolutional decoder [ 5 ] . OLIVE builds on this family of latent masked prediction methods, but adds view augmentation and a waveform reconstruction objective during pre-training for joint analysis and synthesis modeling. View-based self-distillation. View-based self-supervised learning trains representations to agree across stochastic views of the same input [ 47 ] . BYOL showed that an online network can predict the representation of a slowly updated target network without explicit negative pairs, learning invariances through data augmentations [ 23 ] . In audio, BYOL-A adapts this principle using mixup and time–frequency transformations on log-mel spectrograms [ 40 ] , and BYOL-S further studies bootstrapped speech representations across encoder architectures and pre-training data choices [ 19 ] . WaveBYOL extends this approach to raw waveforms using time-domain augmentations, primarily targeting general audio tasks [ 29 ] . Similar to data2vec, WavJEPA predicts latent targets from raw waveforms [ 62 ] , following the broader JEPA framework [ 1 ] . OLIVE builds on this line of view-based self-distillation methods by combining masked latent prediction with time-domain waveform augmentations to explicitly induce invariances, while jointly optimizing a synthesis objective. Speech reconstruction and analysis–synthesis. Recent work trains neural vocoders [ 53 , 43 , 31 , 60 ] such as HiFi-GAN [ 30 ] on frozen self-supervised features, effectively forming neural analysis–synthesis pipelines [ 13 , 42 , 49 , 24 ] , enabling synthesis methods that leverage self-supervised representations. Such methods typically treat synthesis as a downstream process, rather than integrating it into representation learning during pre-training. Neural audio codecs further explore this analysis–synthesis interface by learning compact latent representations that can be decoded back to waveforms [ 63 , 16 , 35 , 11 ] . These works primarily target efficient storage, transmission, or speech language model inference, whereas OLIVE uses reconstruction as a training objective for speech representation learning. Closer to speech SSL pre-training, data2vec-SG augments data2vec with