ConceptCaps: a Distilled Concept Dataset for Interpretability in Music Models

Bruno Sienkiewicz, Łukasz Neumann, Mateusz Modrzejewski · 2026

Read ConceptCaps: a Distilled Concept Dataset for Interpretability in Music Models by Bruno Sienkiewicz, Łukasz Neumann, Mateusz Modrzejewski (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-14157.txt.

Introduction

Bruno Sienkiewicz, Łukasz Neumann, and Mateusz Modrzejewski build a dataset for explaining music AI in 2026. Concept-based interpretability methods like TCAV need clean concept examples. They require well-separated positive and negative examples for each concept. Existing music datasets lack this structure, with sparse, noisy, or ill-defined tags.

Their ConceptCaps holds twenty-one thousand music-caption-tags triplets. Explicit labels come from a two hundred attribute taxonomy. Clean concept data lets researchers test what a music model has learned. Explanations of model taste could build trust in recommendation tools. The set is built for tests, not for play lists. Its clean tags let teams probe one trait at a time.

Catalog record

CREATORBruno Sienkiewicz, Łukasz Neumann, Mateusz Modrzejewski
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-14157

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

ConceptCaps: a Distilled Concept Dataset for Interpretability in Music Models Bruno Sienkiewicz, Łukasz Neumann, Mateusz Modrzejewski Abstract Concept-based interpretability methods like TCAV require clean, well-separated positive and negative examples for each concept. Existing music datasets lack this structure: tags are sparse, noisy, or ill-defined. We introduce ConceptCaps, a dataset of 21k music-caption-tags triplets with explicit labels from a 200-attribute taxonomy. Our pipeline separates semantic modeling from text generation: a VAE learns plausible attribute co-occurrence patterns, a fine-tuned LLM converts attribute lists into professional descriptions, and MusicGen synthesizes corresponding audio. This separation improves coherence and controllability over end-to-end approaches. We validate the dataset through audio-text alignment (CLAP), linguistic quality metrics (BERTScore, MAUVE), and TCAV analysis confirming that concept probes recover musically meaningful patterns. Dataset and code are available online . 1 Introduction Work on interpretability and explainability in music information retrieval has increasingly turned to “concept-level” analyses, such as Testing with Concept Activation Vectors (TCAV 16 ), which offers a promising framework for interpreting these models by identifying and measuring sensitivity to high-level musical aspects like instrumentation, genre, and mood 10 ; 11 ; 1 . In practice, however, these concepts are usually inferred from free-form captions, weak tags, or proxy tasks that do not provide explicit, validated concept labels. This leads to a gap between the theoretical object of analysis (a well-defined musical concept) and the empirical evidence available in existing datasets, which are dominated by abstract, overlapping, and sometimes contradictory descriptions. This gap has concrete consequences. First, it is difficult to construct positive and negative example sets that isolate a concept from closely related phenomena, which undermines the design of controlled experiments. Second, comparisons across models and studies are hard to interpret, because each work often relies on ad hoc concept definitions and bespoke data slices. Third, it is unclear how much of a reported “concept sensitivity” reflects genuine semantic structure in the model, and how much is an artefact of annotation noise or dataset bias. To address these limitations, we propose a novel dataset generation pipeline based on separation of concerns . Similar methods often rely on asking a single LLM to both generate plausible musical attributes and write fluent descriptions. We decompose the task into two specialized stages. First, a Variational Autoencoder (VAE) 18 learns the distribution of musical attribute co-occurrence from curated source data, ensuring that sampled combinations are statistically plausible and musically coherent. Second, a fine-tuned language model translates these controlled attribute lists into professional, context-aware music descriptions. This separation reduces hallucination, improves controllability, enables independent optimization of semantic consistency and linguistic quality, and yields a dataset explicitly designed to support concept-based interpretability analyses. 1.1 Main Contributions The primary contributions of this work are: 1. We introduce ConceptCaps, a large-scale, copyright-free dataset of 21k music-description pairs with rich, validated concept labels and matched counterexamples, specifically designed for concept-based explainability research. 2. We present a novel two-stage generative pipeline that separates VAE-based semantic consistency from fine-tuned LLM linguistic quality, enabling controlled, high-quality music dataset generation while maintaining efficiency through local, reproducible processing. 3. We demonstrate quality and utility using linguistic metrics (BLEU, ROUGE, BERTScore, MAUVE), audio-text alignment (CLAP scores), concept-specific metrics, and downstream TCAV analysis demonstrating both dataset quality and interpretability. 4. We show our approach achieves competitive results with significantly lower computational cost than API-based methods 24 , and provides fine-grained control over dataset characteristics through VAE latent space sampling and conditioning. 2 Related Work 2.1 Concept-Based Explainability in Deep Learning Concept-based methods such as Testing with Concept Activation Vectors (TCAV) 16 have become increasingly popular for interpreting neural networks across vision and audio domains. TCAV enables quantitative measurement of model sensitivity to user-defined concepts by extracting directions in neural network activations that correspond to those concepts. The core insight is that understanding decision-making at the concept level—rather than at the input or neuron level—can provide more intuitive and actionable insights. However, TCAV’s effectiveness depends critically on having clean, well-separated positive and negative concept examples. When concept examples are noisy, sparse, or poorly defined, TCAV scores become unreliable and difficult to interpret. This bottleneck has limited the adoption of concept-based analysis in music research, where high-quality concept datasets are scarce. Our work directly addresses this limitation by providing a systematic, reproducible method to construct such datasets. Related interpretability approaches in audio and music domains have demonstrated the potential of concept-level analysis 9 ; 32 ; 11 ; 10 ; 1 . 2.2 Music Dataset Augmentation and Synthesis Recent work on music captioning has explored using large language models to augment existing music datasets. Approaches such as LP-MusicCaps 8 and WavCaps 22 generate or refine captions using language models, demonstrating the potential of LLM-based data augmentation. While these approaches effectively scale data production, they face two key limitations in the interpretability context: 1. Lack of Controllability: Generated data follows the biases of the LLM’s training distribution rather than explicitly controlling concept patterns needed for concept analysis. 2. Limited Semantic Validation: These methods often rely on existing datasets that can be noisy and lack systematic representation of specific concepts. Without validation of semantic coherence, it is unclear whether generated attributes form meaningful, interpretable patterns. 3 Method Our architecture draws inspiration from successful multi-stage generation approaches in computer vision and other domains, like StackGAN 30 or CLAP 9 . Our pipeline adopts this principle of separation of concerns : the VAE acts as a “Stage-I” generator sketching the semantic skeleton (musical attributes and their plausible combinations), while the fine-tuned LLM acts as a “Stage-II” refiner, constructing descriptions with linguistic detail, professional vocabulary, and contextual nuance. 3.1 Pipeline Overview At a high level, source-derived attributes are first distilled into a consistent taxonomy with representative examples, then modeled as co-occurrence patterns by the VAE. The fine-tuned LLM translates these attribute lists into natural language descriptions, and MusicGen synthesizes corresponding audio. This ensures that each synthetic sample is explicitly paired with concept labels drawn from the learned distribution and representative audio realizations. Figure 1: Overview of the three-stage dataset generation pipeline. Stage 1 (Semantic Modeling via VAE): A Variational Autoencoder samples from the latent space z\sim\mathcal{N}(0,I) and decodes to generate coherent attribute lists (e.g., “folk, acoustic guitar, upbeat”). Stage 2 (Text Generation via Fine-Tuned LLM): The fine-tuned language model (Llama 3.1 8B) receives the attribute list and generates a professional, descriptive music caption capturing the semantic content. Stage 3 (Audio Synthesis): MusicGen synthesizes copyright-free audio from the generated description, completing the dataset sample with well-aligned audio-text pairs and explicit concept labels suitable for interpretability analysis. 3.2 Dataset Distillation & Concept Taxonomy To obtain a high-quality, concept-dense dataset suitable for rigorous interpretability analysis, we must first define a meaningful concept taxonomy and source high-quality training data. The MusicCaps dataset 2 contains 5,521 human-annotated descriptions with associated tags, providing a valuable starting point. However, MusicCaps suffers from sparse and noisy categories. These issues undermine the quality of downstream semantic models. We therefore apply a distillation methodology to ensure our VAE and LLM components receive clean, representative data. The distillation procedure consists of three steps: 1. Tag Extraction and Organization: Identify all unique tags in MusicCaps and group them into coherent semantic categories such as instruments (acoustic guitar, piano, drums), genres (folk, electronic, classical),

Read the complete text