Residual Tokens Enhance Masked Autoencoders for Speech Modeling

Samir Sadok, Stéphane Lathuilière, Xavier Alameda-Pineda · 2026

Read Residual Tokens Enhance Masked Autoencoders for Speech Modeling by Samir Sadok, Stéphane Lathuilière, Xavier Alameda-Pineda (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-19399.txt.

Description

Samir Sadok, Stéphane Lathuilière, and Xavier Alameda-Pineda let a speech model keep what labels leave out. Speech models often lean on named attributes like pitch, content, and speaker identity. Those named parts cannot capture the full richness of a natural voice.

Their framework, RT-MAE, adds trainable residual tokens that soak up whatever the named attributes miss. A masked autoencoder design forces the model to rebuild hidden parts of speech from the rest. Models that admit what their labels miss should sound more like real speech. They should also score more like it, when they rebuild a voice or judge a clip.

Catalog record

CREATORSamir Sadok, Stéphane Lathuilière, Xavier Alameda-Pineda
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE24 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-19399

Browse the full library · About the Tower of Babel