Multimodal Dataset Normalization and Perceptual Validation for Music-Taste Correspondences
Matteo Spanio ( [email protected] )( 0000-0002-2436-7208 ), Valentina Frezzato ( [email protected] ), Antonio Rodà ( [email protected] )( 0000-0001-9921-0590 ) · 2026
Read Multimodal Dataset Normalization and Perceptual Validation for Music-Taste Correspondences by Matteo Spanio ( [email protected] )( 0000-0002-2436-7208 ), Valentina Frezzato ( [email protected] ), Antonio Rodà ( [email protected] )( 0000-0001-9921-0590 ) (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10632.txt.
Introduction
Spanio, Frezzato, and Rodà test whether sound-taste links survive when a small listener-annotated collection is scaled up automatically. Such links appear in museum and food-tourism exhibits. They grow 257 annotated tracks into 49,300 labeled segments from the Free Music Archive. They check the patterns against food-chemistry profiles for 20 largely Italian dishes, rated by 49 listeners. The patterns carry over across every taste dimension. Listener ratings match the computed profiles well beyond chance.
Catalog record
| CREATOR | Matteo Spanio ( [email protected] )( 0000-0002-2436-7208 ), Valentina Frezzato ( [email protected] ), Antonio Rodà ( [email protected] )( 0000-0001-9921-0590 ) |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33842 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.10632
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Multimodal Dataset Normalization and Perceptual Validation for Music-Taste Correspondences Matteo Spanio ( [email protected] )( 0000-0002-2436-7208 ), Valentina Frezzato ( [email protected] ), Antonio Rodà ( [email protected] )( 0000-0001-9921-0590 ) Abstract Collecting large, aligned cross-modal datasets for music–flavor research is difficult because perceptual experiments are costly and small by design. We address this bottleneck through two complementary experiments. The first tests whether audio–flavor correlations, feature-importance rankings, and latent-factor structure transfer from an experimental soundtracks collection (257 tracks with human annotations) to a large FMA-derived corpus ( ∼ \sim 49,300 segments with synthetic labels). The second validates computational flavor targets—derived from food chemistry via a reproducible pipeline—against human perception in an online listener study (49 participants, 20 tracks). Results from both experiments converge: the quantitative transfer analysis confirms that cross-modal structure is preserved across supervision regimes, and the perceptual evaluation shows significant alignment between computational targets and listener ratings (permutation p < 0.0001 p<0.0001 , Mantel r = 0.45 r=0.45 , Procrustes m 2 = 0.51 m^{2}=0.51 ). Together, these findings support the conclusion that sonic seasoning effects are present in synthetic FMA annotations. We release datasets and companion code to support reproducible cross-modal AI research. 1 Introduction Sonic seasoning studies how sound modulates flavor perception, and cross-modal AI offers a route to model these effects at scale [ 1 , 2 ] . The literature documents robust audio–taste correspondences for pitch, timbre, and related musical attributes [ 3 , 4 , 5 , 6 ] , explained by statistical, semantic, and affective mechanisms [ 1 , 7 ] . Controlled studies show measurable perceptual impact on crispness, hedonic judgments, and meal dynamics [ 8 , 9 , 10 ] . Yet progress is constrained by a lack of large, aligned datasets: perceptual experiments use few controlled stimuli and participants, limiting deep-learning and large-sample analyses. Multimodal pipelines also suffer from inconsistent feature spaces, weak normalization across sources, and limited reproducibility. We address these issues through two complementary experiments. The first analyzes whether audio–flavor relationships transfer from an experimental soundtracks collection with human annotations to a large FMA-derived corpus ( \sim 49{,}300 30-second segments) with synthetic labels [ 11 , 2 ] . The second validates computational flavor targets—derived from food chemistry—against human perception in an online listener study. Our central claim is that both quantitative transfer analysis and perceptual evaluation converge to show that sonic seasoning effects are preserved in synthetic annotations. Methodologically, the work contributes to reproducible multimodal science in three ways: (i) it tests consistency of audio–flavor structure across supervision regimes (human labels vs weak model labels); (ii) it provides an independent perceptual validation of computationally derived targets; and (iii) it links dataset-level statistics to perceptual-level outcomes, supporting stronger external validity claims than single-dataset analyses. These choices are especially relevant for sonic seasoning, where perceptual plausibility and interpretability often outweigh maximizing a single predictive metric, and where transferability across supervision regimes remains a practical bottleneck. Main contributions. 1) A quantitative analysis showing that audio–flavor correlations, feature rankings, and latent structure transfer from human-annotated to synthetically labeled data; 2) A perceptual evaluation framework that converts food chemistry into computational targets, matches them to music, and validates the correspondence with a listener study; 3) Evidence that both experiments converge, supporting the presence of sonic seasoning effects in synthetic FMA annotations; 4) Public release of datasets and companion code to support replication and reuse. Research questions. 1) RQ1 (Experiment 1): Do audio–flavor relationships transfer from the experimental soundtracks collection to the FMA-extended corpus with synthetic annotations? 2) RQ2 (Experiment 2): Do computational flavor targets, derived from food chemistry and matched to FMA tracks, align with human perceptual ratings? 2 Related Work 2.1 Cross-Modal and Neuroscientific Background Prior work in multisensory perception shows systematic interactions between auditory cues and gustatory judgments, motivating computational sonic seasoning as a data-driven extension of cross-modal correspondences [ 1 ] . Classic and contemporary studies converge on robust non-arbitrary mappings between auditory and non-auditory features. Early cross-modal findings, including shape-sound correspondences, established that humans consistently align sensory dimensions across modalities [ 12 ] . Later work showed similar regularities for taste and sound, including pitch-taste and timbre-taste associations [ 3 , 4 , 5 ] . Two limitations recur: scale (controlled studies cover limited musical diversity) and representational compatibility (task-specific ratings are hard to propagate across datasets). Without explicit normalization, cross-study comparability remains weak, motivating our target-construction protocol and consistent target semantics. 2.2 Cross-Modal AI and Synthetic/Fused Datasets Recent AI approaches combine neural models, metadata fusion, and synthetic or weakly supervised labels to scale cross-modal tasks. In music–flavor contexts, this enables larger datasets but introduces normalization and validation challenges that require explicit evaluation protocols [ 2 , 13 , 11 ] . The trend is consistent with broader multimodal AI: text-to-image and text-to-music systems demonstrate the utility of large heterogeneous corpora but also expose sensitivity to dataset construction choices [ 14 , 15 ] . In affective and emotionally conditioned modeling, multimodal fusion improves robustness, yet depends heavily on aligned labels and modality-specific preprocessing [ 16 , 17 , 18 ] . In music information retrieval, data-driven feature extraction and transformer-based audio models have improved representational power for downstream prediction tasks. However, cross-modal tasks remain constrained by supervision quality and target reliability. Prior work has shown feasibility for music-taste prediction [ 2 , 19 ] , but reproducible protocols for normalization and cross-regime validation are still limited. A related challenge concerns synthetic and fused datasets. Weak labels can greatly increase scale but may introduce regime shifts; transfer analysis is therefore a primary evaluation objective. Synthetic multimodal datasets in vision–language (e.g., CLEVR, GQA, SMMQG) show how controlled generation can scale supervision and stress-test compositional reasoning [ 20 , 21 , 22 ] . Overall, the literature suggests a gap between high-capacity model development and robust multimodal dataset methodology. This paper addresses that gap by centering normalization, transfer diagnostics, and perceptual validation in a single integrated workflow, rather than proposing a new model architecture. 3 Methodology 3.1 Data Sources and Harmonization We jointly analyze: (i) an experimental soundtracks collection (257 tracks) with human flavor annotations aggregated from 22 published studies, (ii) an FMA-extended chunk-level dataset ( \sim 49{,}300 30-second segments) with synthetic flavor annotations, and (iii) FoodDB-derived food chemistry resources [ 23 ] for target construction. All components are mapped to a common five-dimensional taste space (sweet, bitter, sour, salty, spicy). The FMA flavor labels were generated by an Audio Spectrogram Transformer (AST) [ 24 ] fine-tuned on the experimental soundtracks collection, as described in [ 2 ] . The AST was trained on the 257-track corpus; the transfer analysis in Experiment 1 therefore tests whether audio-feature–flavor structure (correlation profiles, importance rankings, latent factors) is preserved in model-generated labels, not whether the labels are independently derived. No FMA audio was used during AST training, and no human-annotated tracks appear in the FMA corpus, ruling out direct data leakage. Both corpora share a common set of 92 audio features extracted with librosa [ 25 ] from 11 feature families: zero crossing rate (ZCR), chroma (CQT, STFT, CENS), tonnetz, MFCCs (20 coefficients), RMS energy, spectral centroid, spectral bandwidth, spectral contrast (7 sub-bands), and spectral rolloff. For each time-varying feature, mean and standard deviation are