CAK: Emergent Audio Effects from Minimal Deep Learning

Austin Rockman · 2025

Read CAK: Emergent Audio Effects from Minimal Deep Learning by Austin Rockman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02643.txt.

Introduction

Austin Rockman coaxes audio effects out of a single tiny convolutional kernel, in work from 2025. The kernel holds just nine learned weights, trained on two hundred samples from a personal sound corpus. Two tricks make it work. The output adds a learned pattern scaled by a control knob, with a soft gate that returns the clean signal at zero. A flipped adversarial game has networks check that the requested control value was truly applied.

Instead of judging real from fake, the partners cooperate to verify the setting, and odd transformations emerge. The learned kernel settles into a diagonal form that shifts frequencies in time, yielding musical effects shaped by the input itself.

Catalog record

CREATORAustin Rockman
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02643

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CAK: Emergent Audio Effects from Minimal Deep Learning Austin Rockman Abstract We demonstrate that a single 3×3 convolutional kernel can produce emergent audio effects when trained on 200 samples from a personalized corpus. We achieve this through two key techniques: (1) Conditioning Aware Kernels (CAK), where output = input + (learned_pattern × control), with a soft-gate mechanism supporting identity preservation at zero control; and (2) AuGAN (Audit GAN), which reframes adversarial training from "is this real?" to "did you apply the requested value?" Rather than learning to generate or detect forgeries, our networks cooperate to verify control application, discovering unique transformations. The learned kernel exhibits a diagonal structure creating frequency-dependent temporal shifts that are capable of producing musical effects based on input characteristics. Our results show the potential of adversarial training to discover audio transformations from minimal data, enabling new approaches to effect design. 1 Introduction Generative AI has captured the world’s imagination by transforming deep learning from an analytical tool into a creative medium. While GANs and diffusion models enable visual art and music generation, the manipulation of existing audio remains primarily rooted in traditional signal processing. Mathematical DSP has given us powerful audio effects, from convolution reverbs to analog circuit models, but these rely on human insight to translate acoustic phenomena into equations. What if we could learn audio effects directly from sound itself? We bridge this gap with Conditioning Aware Kernels (CAK), a modulation technique that discovers transformations directly from data through adversarial training. Our approach explores an alternate perspective: neural networks as simplification approximators. By constraining our model to find minimal viable solutions, we investigate whether sophisticated audio transformations can arise from 200 training samples and 11 learnable parameters (a single 3×3 kernel with bias and scale). This framework allows us to study what emerges when model capacity is deliberately limited to match realistic data constraints. Human perception mirrors this efficiency. We learn to recognize complex patterns from a handful of experiences. Similarly, experienced audio engineers develop intuition for effects through limited but focused interaction. CAK captures this principle computationally: given a small corpus of audio with varying features, our system learns not just to reproduce but to discover and apply learned qualities across unseen inputs. Traditional GANs (Goodfellow et al., 2014) pit generator against discriminator in a forgery detection contest. We propose an ’audit game’ (AuGAN) where the discriminator must verify that the generator applied the user’s control value to its learned features, whatever those features turn out to be. This structural shift, from deception to verification, enables the discriminator to guide the discovery of audio transformations. Through CAK, we investigate whether neural transformations require architectural complexity, or whether they can emerge from the interaction between minimal learned operations and the inherent richness of audio signals. 2 Related Work Neural Audio Synthesis and Effects : WaveGAN (Donahue et al., 2018) and GANSynth (Engel et al., 2019) first showed that adversarial training can generate raw waveform audio using large datasets and models. DDSP (Engel et al., 2020) and RAVE (Caillon & Esling, 2021) achieve high quality synthesis through compact architectures and strong inductive biases, enabling efficient training even with limited data. Unlike those works, CAK is not a standard generator; it learns effects from a small, personalized corpus. Conditioning Mechanisms : Feature - wise Linear Modulation (FiLM; Perez et al., 2018) conditions deep networks via channel wise affine transforms. In a 2019 retrospective, the authors note that FiLM often needs additional task - specific inductive biases to remain data efficient (Perez et al., 2019 retrospective), a limitation we also observed when applying FiLM to complex conditioning vectors. Dynamic kernel methods such as CondConv (Yang et al., 2019) synthesize weights by mixing basis filters. CondConv effectively asks which kernel a layer should use; CAK instead detects a salient pattern and modulates its residual contribution by the user’s control value. Few-Shot Learning : Audio few-shot work typically relies on meta - learning (MAML, Finn et al., 2017; Prototypical Networks, Snell et al., 2017), which require access to large and diverse meta-training sets composed of many tasks. CAK operates in a more extreme regime: it learns directly from a single, 50 - minute corpus without episodic sampling or meta - tasks. Emergent Complexity from Simple Rules : Growing Isotropic Neural Cellular Automata (Mordvintsev, Randazzo, and Fouts, 2022) demonstrates similar principles in the visual domain, where simple local update rules produce complex emergent patterns. Like CAK, this work shows that behavioral diversity can arise from the interaction between minimal fixed rules and varying initial conditions, rather than from architectural complexity. Biological Inspiration : Our emphasis on minimal, data - efficient representations echoes the efficient coding hypothesis (Barlow, 1961) and sparse coding results in V1 (Olshausen & Field, 1996) which suggest biological systems seek minimal representations. Feature Integration Theory (Treisman & Gelade, 1980) likewise suggests that selective modulation of simple detectors can explain complex percepts, paralleling CAK’s single - kernel modulation experiment. 3 Method 3.1 Empirical Motivation Our initial approach followed established conditioning methods, using FiLM (Perez et al., 2018) with 24-dimensional control vectors encoding categorical tags, continuous DSP parameters, and perceptual attributes. This exhibited training instability. FiLM’s affine transformations rely on deep networks to achieve complex modulation, making them potentially unsuitable for rich musical descriptors and small datasets. We do not claim that FiLM is fundamentally limited in this regard, only that we could not find the optimal solution for this problem using conventional modulation methods. This failure revealed an insight: complex control may not always require complex modulation. Through ablation, we discovered that our initially complex network consistently relied on a small subset of detected patterns, regardless of the control vector’s dimensionality. This suggested a different approach: instead of learning how to modulate based on complex inputs, learn what patterns to detect, then simply scale them. The audit game naturally enforces this sparsity. The discriminator must verify control values, incentivizing the generator to find distinct patterns. This led to CAK: a single learned detector whose output is scaled by a scalar control value. The shift from 24-dimensional modulation to 1-dimensional scaling did not reduce expressiveness; it revealed that one well-learned pattern could create effects through context-aware application. Figure 1: Training dynamics comparison between FiLM and CAK architectures. Both networks were trained under identical conditions with no divergence mitigation strategies. FiLM-based conditioning with 24-dimensional control vectors exhibits instability, with discriminator and generator loss exceeding 10^{4} and training failure. In contrast, CAK achieves stable training on the same dataset, enabling the discovery of patterns despite its ultimate convergence to a single dominant feature. Instead of fighting against the results of the network, we theorized that dimensional collapse might contain a fascinating study of how neural networks process complex domains. We consider music to be infinitely analyzable. A single sample contains many possible interpretations. Collapse is usually treated as a failure to be mitigated through architectural tricks or regularization, and in some cases may be the correct motive, but perhaps we could design architectures that embrace, rather than resist, this tendency toward simplification. 3.2 CAK Architecture The core CAK operation implements a simple principle: \text{output = input + (learned\_pattern}\times\text{control)} Formally, this becomes: y=x+(D(x)\times c\times\sigma(c)\times s) (1) where: • x\in\mathbb{R}^{F\times T} is the

Read the complete text