PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association
Abdul, Muhammad Arslan, Shah, Muhammad Irzam, Markus, Mubashir · 2025
Read PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association by Abdul, Muhammad Arslan, Shah, Muhammad Irzam, Markus, Mubashir (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-17002.txt.
Introduction
Abdul Hannan and co-authors matched faces to voices with a new method in two thousand twenty-five. People can often guess a face from a voice roughly. Teaching machines that skill aids search and media tools. Past methods leaned on careful mining of negative pairs. They also hinged on a sensitive margin setting. The new method, PAEFF, aligns the two data types first.
Face and voice embeddings live in unlike spaces. Precise alignment brings them into a shared frame. A gated fusion then blends them with learned weights. Right-angle constraints keep fused codes clean and separate. Tests on a celebrity video dataset showed clear gains. The authors say careful fusion beats brute-force mining. The method may help verify speakers in video calls.
Catalog record
| CREATOR | Abdul, Muhammad Arslan, Shah, Muhammad Irzam, Markus, Mubashir |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-17002
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
PAEFF: Precise Alignment and Enhanced Gated Feature Fusion for Face-Voice Association Abdul, Muhammad Arslan, Shah, Muhammad Irzam, Markus, Mubashir Abstract We study the task of learning association between faces and voices, which is gaining interest in the multimodal community lately. These methods suffer from the deliberate crafting of negative mining procedures as well as the reliance on the distant margin parameter. These issues are addressed by learning a joint embedding space in which orthogonality constraints are applied to the fused embeddings of faces and voices. However, embedding spaces of faces and voices possess different characteristics and require spaces to be aligned before fusing them. To this end, we propose a method that accurately aligns the embedding spaces and fuses them with an enhanced gated fusion thereby improving the performance of face-voice association. Extensive experiments on the VoxCeleb dataset reveals the merits of the proposed approach 1 1 1 Our code is available at: https://github.com/hannabdul/paeff . Hannan Manzoor Nawaz Liaqat Schedl Noman University of TrentoItaly Mohamed bin Zayed University of Artificial IntelligenceU.A.E. Johannes Kepler UniversityAustria IMT LuccaItaly Linz Institute of Technology, AI LabAustria keywords Multimodal learning, Face-voice association, Cross-modal verification & matching, Hyperbolic Space. 1 Introduction Face-voice association is a widely studied task in cognitive science, which investigates the relationship of the human faces with their voices [ 1 , 2 , 3 ] . This task is brought into the computer vision community by Nagrani et al. [ 4 ] by formulating a deep learning approach to identify which face pairs belong to the voice segment. Consequently, several methods [ 5 , 6 , 7 , 8 , 9 , 10 , 11 , 12 , 13 ] have been proposed to tackle this problem. These methods rely on unimodal encoders to separately extract the face and voice embeddings followed by learning the discriminative space which minimizes the distance between the embeddings of same identities. Typically, contrastive or triplet loss formulations are leveraged to achieve this task [ 5 , 6 , 4 , 14 , 15 ] . Although demonstrating favorable performance in learning discriminative space, such loss formulations are limiting in the sense that they require tuning of a margin hyperparameter which is challenging as the distances between samples can differ significantly during training. Furthermore, the training complexities of contrastive and triplet losses are O(n^{2}) and O(n^{3}) , respectively (where n is the number of samples for a face or voice) resulting in longer training time as the size of the dataset increases dramatically. To this end, several methods have explored the utilization of auxiliary identity information as an alternative to contrastive or triplet loss formulations. For example, Nawaz et al. [ 8 ] introduced an approach to learn the discriminative embedding space by leveraging identity centers, eliminating the need of pairwise or triplet supervision. These methods are fairly effective, and require the necessity of being utilized with traditional classification losses such as cross-entropy (CE) loss, resulting in counterintuitive combination of Euclidean and angular spaces respectively. To mitigate this, Saeed et al. [ 10 ] have proposed the fusion and orthogonal projection (FOP) method that introduces a loss by imposing orthogonality constraints on the fused embeddings of faces and voices to learn discriminative embedding space. FOP performs favorably on the face-voice association task, however, fusing the features of two modalities without aligning the corresponding embedding spaces restricts the performance. In this work, we hypothesize that the fusion of feature representations that come from different embedding spaces is ineffective unless the two embedding spaces are aligned, and it limits the ability of loss formulation to maximize the distance between the inter-identity representations. Therefore, a mechanism is required that can align the embedding spaces of two modalities before fusing them. To this end, we propose a dual branch face-voice association framework that enables the model to align the embeddings of faces and voices in hyperbolic space by preserving the distances and complex relationships before fusing them. In addition, we propose a fusion module that can effectively fuse the feature representations resulting in performance improvement of face-voice association tasks. In summary, our contributions are as follows. • We propose a dual branch face-voice association framework that precisely aligns the multimodal feature embeddings before merging them. • We introduce an effective feature fusion strategy that utilizes the attention weights to merge face and voice feature embeddings, thereby improving model performance. • We perform extensive experiments on a challenging and benchmark face-voice association dataset (VoxCeleb [ 16 ] ) to demonstrate the effectiveness of the proposed framework. Figure 1: (a) Overall illustration of the proposed face-voice association approach. Face ( X_{f} ) and voice ( X_{v} ) features are extracted by utilizing vision and audio encoders, respectively. Extracted features X_{f} and X_{v} are then fed to linear layers to obtain the projected features of dimension D . The projected features are transformed to hyperbolic space ( \mathbb{H}_{2} ) for accurate alignment of feature representations. Symmetric cross-entropy loss L_{A} is utilized to align the feature representations. Afterwards, aligned features are fused by using enhanced gated feature fusion (EGFF) module. The fused features are fed to the logits layer and optimized by means of orthogonal projection ( L_{OP} ) and cross-entropy ( L_{CE} ) losses. (b) Detailed overview of EGFF is shown on the right side. AW refers to the computation of attention weights for feature fusion. 2 Background This section summarizes previous work related to multimodal learning, face-voice association and hyperbolic embedding space. Multimodal learning and face-voice association. Multimodal learning has been an important research area in recent years due to the fact that information in real-world scenarios comes through multiple and diverse data sources [ 17 , 18 , 19 , 20 , 21 ] . For instance, human behaviors, emotions, actions, and humor consist of multiple modalities such as speech, spoken language, and faces, thus various human-centered tasks are extensively studied including face-voice association [ 6 ] . Nagrani et al. [ 4 , 5 ] illustrated that the distance between face and voice embeddings can be minimized by using either contrastive or triplet loss formulation to learn face-voice association. Later, Nawaz et al. [ 8 ] proposed a method to learn a discriminative embedding space by utilizing identity centers, removing the requirement for pairwise or triplet supervision. More recently, Saeed et al. [ 10 ] utilized the fused embeddings and employed the orthogonality constraints to minimize the distance between the feature embeddings of same classes. Since face and voice feature representations belong to different embedding spaces, therefore, we propose to align these embeddings to obtain discriminative embedding space. Hyperbolic Space. The aim of embedding a space into another requires the preservation of complex relationships of the feature representations. Sala et al. [ 22 ] showed that hyperbolic space can better preserve distances and complex relationships than Euclidean space. The authors explained how hyperbolic spaces are related to Poincare disk \mathbb{H}_{2} that contains all the points in a unit disk in two dimensions. Later, Khrulkov [ 23 ] showed that hyperblic embeddings are suitable for various computer vision tasks including image classification, image retrieval, and few-shot learning. More recently, hyperbolic space is utilized for various multimodal tasks [ 24 , 25 ,