Multi-Axis Speech Similarity via Factor-Partitioned Embeddings
Jim, Jens · 2026
Read Multi-Axis Speech Similarity via Factor-Partitioned Embeddings by Jim, Jens (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-02804.txt.
Introduction
Jim O'Regan and Jens Edlund build speech embeddings that keep separate traits in separate lanes of one vector. A voice carries content, identity, dialect, and gender at once, but a standard embedding mixes them into a single point, so similarity searches cannot say which trait should count. Their framework gives each trait its own subspace.
A shared acoustic encoder feeds a small projection head per trait. Each head is trained by distillation from a specialist teacher or by contrast over pairs sharing a label. Retrieval then scores each lane apart and adds them with signed weights, so a search can stress what was said, stress how it sounded, or suppress one trait to surface another. On cross-corpus tests with shared sentence prompts, signed weighting reduced same-speaker bias and found matching content across recording conditions.
Catalog record
| CREATOR | Jim, Jens |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.02804
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multi-Axis Speech Similarity via Factor-Partitioned Embeddings Jim, Jens Abstract Speech encodes multiple simultaneous attributes–linguistic content, speaker identity, dialect, gender–that conventional single-vector embeddings conflate. We present a factor-partitioned embedding framework that maps each utterance into a single vector whose subspaces correspond to distinct axes of variation. A shared acoustic encoder feeds per-axis linear projection heads, each trained via distillation from a specialist teacher or a contrastive objective over shared-label pairs. The resulting embeddings support attribute-conditioned retrieval: similarity is computed as a signed weighted sum over per-axis cosine scores, allowing retrieval that jointly considers what was said and how–or explicitly suppresses one attribute to surface another. We evaluate on cross-corpus retrieval over corpora sharing the Harvard sentence prompts, demonstrating that signed axis weighting can suppress same-speaker bias and surface semantically matched utterances across recording conditions. Code is available at: https://github.com/jimregan/spoken-sentence-transformers O’Regan Edlund \interspeechcameraready KTH Royal Institute of TechnologySweden keywords speech similarity, multi-attribute retrieval 1 Introduction Speech, in contrast with text, is inherently variable. We never say the same thing the same way twice: different speakers say the same thing differently, and even the same speaker says the same thing differently at different times, due to a variety of factors such as mood, context, environment, and communicative intent. Beyond linguistic content, speech reveals much about the speaker: more fundamentally, it can reveal their identity, sex, and even health [ 22 ] . These attributes are often intertwined, and it can be difficult to disentangle them. The prompts presented to speakers when creating text-to-speech datasets have traditionally been specifically selected to be phonetically balanced [ 18 , 28 ] . In multi-speaker datasets, there is often substantial overlap between the prompts read by different speakers. This overlap presents an opportunity to evaluate the feasibility of attribute-conditioned similarity: given two utterances of the same sentence, an ideal representation should allow retrieval based not only on what was said, but who said it. We introduce a factor-partitioned embedding framework for speech that learns a single embedding space, partitioned into subspaces corresponding to different attributes. Given an utterance, the model produces a concatenated embedding vector whose subregions are specialised for different factors, such as semantic content and speaker identity. Similarity in this framework is treated as a vector instead of a scalar: instead of computing a single similarity score between utterances, we compute per-axis similarities and combine them with a weighted sum. Retrieval can then be conditioned on different attributes: similarity can be computed jointly over multiple axes, or an attribute can be suppressed. For example, assigning a negative weight to the speaker identity axis discourages same-speaker matches, and surfaces similar results from different speakers. While speech embeddings have been studied extensively, most approaches produce a single similarity measure that conflates multiple attributes. To our knowledge, controllable multi-axis similarity search for speech has not previously been explored. 2 Related work 2.1 Speech representation learning Self-supervised speech models, such as wav2vec 2.0 [ 2 ] , HuBERT [ 14 ] , and WavLM [ 4 ] adapt the masked language model pre-training objective introduced by BERT [ 8 ] to learn general purpose representations for speech. These models capture a mixture of linguistic, speaker, and prosodic information within a single representation; while this allows a single shared model to be fine-tuned on a variety of tasks [ 33 ] , it also makes it difficult to control which attributes drive downstream similarity. Speaker embeddings such as x-vectors [ 27 ] address this for speaker recognition specifically, but do not carry semantic content. Acoustic word embeddings (AWEs) map fixed-length audio segments to vectors such that phonetically similar segments are nearby, enabling query-by-example spoken term detection without transcription [ 17 , 10 , 5 ] . The retrieval evaluation paradigm–retrieve audio items matching a spoken query–motivates the cross-corpus setup used here, albeit at the utterance level rather than the word or segment level. AWE models optimise a single notion of acoustic similarity; our work extends this to multiple simultaneous axes, to allow retrieval to be steered toward semantic content, speaker identity, or other properties independently. Sentence-BERT [ 25 ] adapts BERT-family models to produce fixed-size sentence embeddings suitable for semantic similarity via contrastive training 1 1 1 Oord, Aaron van den, Yazhe Li, and Oriol Vinyals. 2018. “Representation Learning with Contrastive Predictive Coding.” arXiv Preprint 1807.03748. . The SentenceTransformers 2 2 2 https://github.com/huggingface/sentence-transformers library generalises this pipeline to arbitrary encoder–pooler–projector stacks. We adapt this pipeline to speech by replacing the text encoder with an acoustic encoder (WavLM or similar) and introducing multiple attribute-specific projection heads. Conditional Similarity Networks [ 29 ] are the closest conceptual prior for our retrieval approach: they learn a single embedding space partitioned into subspaces corresponding to different kinds of similarity, and select or re-weight subspaces at query time via a mask or weight vector. Our work extends this idea to speech in three respects: (1) application to the speech modality, (2) signed weighting rather than binary masking, and (3) axis supervision via teacher distillation rather than triplet supervision. The semantic axis is an instance of speech-to-text retrieval alignment: the speech encoder is trained to produce embeddings that agree with a text-side teacher, enabling content-based retrieval without relying on ASR at inference time. This pattern has been explored in SpeechDPR [ 19 ] and SpeechRAG [ 21 ] , which align speech encoders to frozen text dense retrievers. CLAP [ 9 ] takes a related approach at the audio level, training a joint audio–text embedding space via contrastive learning over audio clips paired with natural-language descriptions. 2.2 Disentangled speech representations Our work has a degree of superficial similarity to work on disentangled representation learning; our goals, however, are more modest. Rather than enforcing factor independence within the encoder, we focus on supervised learning of multiple relational axes and train per-axis projection heads that induce distinct similarity geometries over a shared frozen acoustic representation space. The earliest work on sequential disentanglement used factorised hierarchical variational autoencoders (FHVAEs) [ 15 ] , which separate sequence-level factors, such as speaker identity, from segment-level factors, such as phonetic content, via time-scale structure without any factor labels. This established the foundational insight that content and speaker information operate on different time scales and can be separated structurally. SpeechSplit [ 23 ] frames this decomposition explicitly: “Human speech conveys a rich stream of information, which can be roughly decomposed into four important components: content, timbre, pitch and rhythm.” These four components map naturally onto axes that a retrieval system might wish to control independently. SpeechSplit separates them using an information bottleneck that restricts the bandwidth available to each encoder branch. ContentVec [ 24 ] incorporated speaker disentanglement directly into the SSL pre-training objective, showing that masked prediction combined with speaker conditioning can yield representations where the content axis carries substantially less speaker information. [ 20 ] extend content/timbre separation to three factors by adding prosody, using a VAE with restricted channel capacity to force disentanglement without explicit supervision. Their approach is generative and unsupervised; ours is discriminative and supervision-guided, using pre-trained teacher models to provide independent training signal for each axis. The key difference: they restrict information flow structurally via \beta\text{-VAE} [ 13 ] bottleneck; we provide axis-specific objectives derived from teachers