An Event-based Sequence Modeling Approach to Recognizing Non-Triad Chords with Oversegmentation Minimization

Leekyung Kim, Jonghun Park · 2026

Read An Event-based Sequence Modeling Approach to Recognizing Non-Triad Chords with Oversegmentation Minimization by Leekyung Kim, Jonghun Park (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-24386.txt.

Introduction

Kim and Park recast auto chord recognition as segment-level sequence prediction. Instead of tagging frames one by one, their model predicts chord sequences auto-regressively, spotting changes only at segment edges. This design cuts extra segmentation, a key weakness alongside scarcity and skew, especially for complex non-triad chords rare in existing datasets. Two new token codes and encoder pre-training, both built for time-aligned chord work, further support the approach. Tests show gains in chord recognition and segmentation, with lifts for complex and rare chord types.

Catalog record

CREATORLeekyung Kim, Jonghun Park
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17157
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.24386

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

An Event-based Sequence Modeling Approach to Recognizing Non-Triad Chords with Oversegmentation Minimization Leekyung Kim, Jonghun Park Abstract Automatic chord recognition (ACR) extracts time-aligned chord labels from music audio recordings. Despite recent advances, ACR still struggles with oversegmentation, data scarcity, and imbalance, especially in recognizing complex chords such as non-triads, which are unpopular in existing datasets. To address these challenges, we reformulate ACR as a segment-level sequence-to-sequence prediction task, where chord sequences are predicted auto-regressively rather than frame by frame. This design mitigates excessive segmentation by detecting chord changes only at segment boundaries. We further introduce two types of token representations and an encoder pre-training method, both specifically designed for time-aligned chord modeling. Experimental results show that our model improves performance in both chord recognition and segmentation, with notable gains for complex and infrequent chord types. These findings demonstrate the effectiveness of segment-level sequence modeling, structured tokenization, and representation learning for advancing chord recognition systems. † † address: 1 Department of Industrial Engineering and Institute of Industrial Systems Innovation, Seoul National University, Seoul, Republic of Korea 1 Introduction Figure 1: Comparison of (a) frame-level classification and the proposed token representations: (b) MERGE and (c) SPLIT. Automatic Chord Recognition (ACR) is the task of identifying the sequence of chords in musical audio, where chords—a group of simultaneously played notes—form the harmonic foundation of a musical piece [ 21 ] . ACR remains an active area of research due to complex musical structures and diverse instrumentation, which make chord annotation time-consuming and demand highly specialized musical expertise. Transformer [ 23 ] -based models have recently achieved state-of-the-art performance in ACR [ 20 , 3 , 4 ] , leveraging their success in natural language processing [ 7 ] . BTC [ 20 ] employed an encoder-only Transformer for audio input, while Harmony Transformer (HT) [ 3 , 4 ] adopted a full encoder-decoder structure, performing segmentation with the encoder and recognition with the decoder on symbolic input. In this work, we focus on ACR from audio, which is more applicable to real-world music scenarios. Recent studies have expanded the target vocabulary from 25 basic chords to a broader range of chord types [ 5 , 9 ] , including non-triad chords. However, such complex chords are unpopular in datasets, leading to data imbalance and limited generalization. To address these issues, prior works have explored structured chord representations [ 9 , 16 ] , even chance training [ 6 ] , curriculum learning [ 22 ] , semi-supervised learning [ 1 , 14 ] , and adaptive loss functions [ 1 ] . Despite these advances, deep learning-based ACR methods still face three major challenges. (1) Oversegmentation: In traditional frame-level classification, probabilities are predicted at every time frame, making them highly sensitive to small fluctuations. A single misclassified frame can split a chord segment and cause excessively fragmented and unstable boundaries [ 9 ] , as shown in Fig. 1 (a). By contrast, our approach predicts chords at segment level, detecting chord changes only when they occur, thereby mitigating oversegmentation. To this end, we reformulate ACR as a segment-level sequence-to-sequence (seq2seq) learning problem and introduce a Transformer encoder-decoder that predicts chord sequences auto-regressively. (2) Data scarcity: The difficulty of chord annotation and copyright restrictions limit the availability of large-scale labeled datasets [ 18 ] . (3) Data imbalance: The skewed chord distribution biases models toward common chords, while underrepresenting unpopular or complex ones [ 19 ] . To address these data-related issues, we propose two types of token representations—MERGE and SPLIT—and an encoder pre-training method based on chord similarity. The contributions of this work are as follows: • We reformulate ACR as a segment-level seq2seq task and propose an auto-regressive chord prediction framework using a Transformer encoder-decoder architecture. To the best of our knowledge, this is the first study to apply this approach to ACR. • We introduce two token representations for time-aligned chord modeling and a chord similarity-based encoder pre-training strategy to address data scarcity and imbalance. • We conduct experiments demonstrating that our model 1 1 1 https://github.com/KimLeekyung/ACR_seq2seq achieves state-of-the-art performance in both chord recognition and segmentation. 2 Proposed Method 2.1 Problem Specification Most traditional ACR models predict the probability of chords at each time frame from an input spectrogram X_{\text{spec}}\in\mathbb{R}^{N_{T}\times N_{F}} , where N_{T} is the number of time frames and N_{F} is the number of frequency bins. This can be formulated as {\hat{y}_{i}}=\arg\max_{y_{i}\in\mathcal{V}}P(y_{i}|X_{{\text{spec}}})\mbox{ for }i=1,\dots,N_{T} , where \mathcal{V} is a pre-defined chord vocabulary, and \hat{y}_{i} is the predicted chord for the i^{th} time frame, selected as the class with the highest probability among y_{i}\in\mathcal{V} . This paper models ACR as a seq2seq problem that auto-regressively predicts token sequence from audio sequence, which is formulated as {\hat{\sigma}_{j}}=\arg\max_{\sigma_{j}\in{\Sigma}}P(\sigma_{j}|X_{{\text{spec}}},\sigma_{1:j-1}) \mbox{for }j=2,\dots , where \sigma_{j} is the j^{th} token in a token sequence and \Sigma is a fixed token set. A predicted token \hat{\sigma}_{j} is the one with the highest probability, given the spectrogram and the previous token sequence \sigma_{1:j-1} . The first token, \sigma_{1} , is fixed as the start-of-sequence token, and the token prediction continues until the end-of-sequence token is predicted or the token sequence reaches the maximum length. 2.2 Model Structure and Chord Representation Figure 2: Overall architecture of the Transformer encoder-decoder model. N_{enc} and N_{dec} denote the numbers of encoder and decoder layers, respectively. This paper adopts the Transformer encoder-decoder structure due to its improved performance and suitability for auto-regressive sequence prediction, as illustrated in Fig. 2 . Audio segments of T seconds are first converted into waveforms and then transformed into spectrograms. To reduce excessive segmentation [ 12 ] and eliminate the need for additional modules such as conditional random fields [ 13 ] often required to smooth predictions in the traditional methods [ 9 , 11 ] , we propose two types of token representations, MERGE and SPLIT, illustrated in Fig. 1 (b) and Fig. 1 (c), respectively. In MERGE, each time-aligned chord is expressed using a time token indicating the chord boundary and a chord token representing the full chord from \mathcal{V} . As \mathcal{V} includes ‘no chord’ and ‘unknown chord’, chord durations are implicitly defined as continuing until the next chord onset. SPLIT decomposes each chord token into root and quality tokens. For instance, C:maj is divided into the root C indicating the chord’s fundamental note, and the quality maj , describing its interval structure. This decomposition enables multiple chords, such as C:dim and B:dim , to share the same quality token, dim . As a result, SPLIT increases the amount of training data available for unpopular chord qualities and facilitates the learning of structural similarities across chords, thereby improving the robustness of learning for unpopular chord qualities. The token sets are defined as follows: \Sigma_{\text{M}}=\mathbb{T}\cup\mathbb{C}\cup\mathbb{S} for MERGE, and \Sigma_{\text{S}}=\mathbb{T}\cup\mathbb{R}\cup\mathbb{Q}\cup\mathbb{S} for SPLIT, where \mathbb{T} , \mathbb{C} , \mathbb{R} , \mathbb{Q} , and \mathbb{S} denote the sets of time, chord, root, quality, and special tokens, respectively. The time token specifies absolute time within each segment at 0.1-second intervals. The chord token in MERGE represents each chord from \mathcal{V} . The root token includes all 12 pitch classes and a shared token for ‘no chord’ and ‘unknown chord’. The quality token spans all chord qualities in \mathcal{V} ,

Read the complete text