Cross-Attention Message-Passing Transformers for Code-Agnostic Decoding in 6G Networks
Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, Jong-Seon No · 2025
Read Cross-Attention Message-Passing Transformers for Code-Agnostic Decoding in 6G Networks by Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, Jong-Seon No (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-01038.txt.
Introduction
Seong-Joon Park and four co-authors build a 2025 AI decoder that can read many error-correcting codes without retraining. Future 6G networks must serve mixed needs, and classic decoders are built code by code. That one-by-one build lacks flex and scale. A single learned decoder for all codes is the goal.
Their model swaps messages between two views of the received signal through masked attention blocks. Scores topped other single neural decoders. A stretched version stays unchanged across code length, rate, and class, so one trained model serves many codes. An ensemble version, running twin blocks with unlike check tables, lifts short-code scores most. The authors pitch the design as a base model for 6G coding.
Catalog record
| CREATOR | Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, Jong-Seon No |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 53 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-01038
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cross-Attention Message-Passing Transformers for Code-Agnostic Decoding in 6G Networks Seong-Joon Park, Hee-Youl Kwak, Sang-Hyo Kim, Yongjune Kim, Jong-Seon No Abstract Channel coding for 6G networks is expected to support a wide range of requirements arising from heterogeneous communication scenarios. These demands challenge traditional code-specific decoders, which lack the flexibility and scalability required for next-generation systems. To tackle this problem, we propose an AI-native foundation model for unified and code-agnostic decoding based on the transformer architecture. We first introduce a cross-attention message-passing transformer (CrossMPT). CrossMPT employs two masked cross-attention blocks that iteratively update two distinct input representations–magnitude and syndrome vectors–allowing the model to effectively learn the decoding problem. Notably, our CrossMPT has achieved state-of-the-art decoding performance among single neural decoders. Building on this, we develop foundation CrossMPT (FCrossMPT) by making the architecture invariant to code length, rate, and class, allowing a single trained model to decode a broad range of codes without retraining. To further enhance decoding performance, particularly for short blocklength codes, we propose CrossMPT ensemble decoder (CrossED), an ensemble decoder composed of multiple parallel CrossMPT blocks employing different parity-check matrices. This architecture can also serve as a foundation model, showing strong generalization across diverse code types. Overall, the proposed AI-native code-agnostic decoder offers flexibility, scalability, and high performance, presenting a promising direction to channel coding for 6G networks. I Introduction The sixth generation (6G) of wireless communication is envisioned to support an unprecedented range of scenarios, from hyper-reliable low-latency communication (HRLLC) to massive communication [ 2 , 3 ] . This expands upon the 5G standard, which already employs different coding schemes like low-density parity-check (LDPC) and polar codes for different use cases [ 4 , 5 ] . Continuing this trend for each diverse 6G scenario would require a growing number of specialized codes and their corresponding decoders, which complicates system integration and increases hardware overhead [ 5 ] . This broad range of scenarios imposes diverse requirements on the physical layer, particularly on channel coding, and calls for a highly flexible and adaptable decoding approach [ 5 , 6 ] . To meet these challenges, a unified and code-agnostic decoder capable of handling various code types, lengths, and rates is essential for enabling flexible and scalable channel coding in 6G networks [ 4 , 5 ] . However, most decoders are designed for specific codes with fixed parameters, limiting their flexibility [ 5 ] . While some algorithms can decode multiple code classes and are considered universal decoders, they still require separate implementations for different code structures. This limitation highlights the need for a foundation model in error correcting code (ECC) decoding; one that can generalize across diverse codes without requiring code-specific adaptation. Enabled by recent advances in deep learning techniques [ 7 , 8 , 9 , 10 ] , AI-based ECC decoders can learn complex representations and generalize across a wide range of code structures, paving the way for a foundation model for ECC decoding. I-A Main Contributions In this paper, we propose an AI-native, unified, and code-agnostic decoder using the transformer architecture. We first propose the cross-attention message-passing transformer (CrossMPT) that employs cross-attention modules to emulate the message-passing decoding algorithm. Our CrossMPT effectively updates the magnitude and syndrome of received vectors iteratively using two masked cross-attention blocks. These blocks utilize the parity-check matrix (PCM) and its transpose as attention masks, guiding the model to focus exclusively on connected node interactions defined by the code structure. As in traditional message-passing decoders such as belief propagation (BP) [ 11 ] or min-sum (MS) algorithm [ 12 ] , where messages are exchanged only between connected variable and check nodes, the masked cross-attention mechanism ensures that the attention scores are computed only over valid edges specified by the PCM. By integrating the transformer’s powerful attention mechanism into the message-passing decoding principle, our CrossMPT has achieved state-of-the-art decoding performance among single neural decoders. Furthermore, we propose a foundation model of CrossMPT (FCrossMPT) by replacing the code-dependent parameters of CrossMPT with code-invariant embeddings. This design makes the model fully agnostic to code-specific properties such as code class, length, and rate, enabling it to be trained on diverse codes with varying configurations. As a result, a single trained model can effectively decode multiple types of codes without retraining or architectural modifications. To boost the decoding performance of short blocklength codes, we propose CrossMPT ensemble decoder (CrossED)–an ensemble decoding architecture that consists of multiple cross-attention modules, each employing a distinct attention mask derived from a different PCM. Ensemble decoding is well-aligned with the requirements of 6G networks, particularly HRLLC, as it enhances decoding accuracy through diversity while maintaining low decoding latency through parallel processing. This is especially critical in short blocklength regimes, where conventional decoders often face performance limitations under strict latency constraints. Our ensemble design leverages complementary attention masks generated by cyclically shifting the systematic PCM, allowing each decoder to attend to different structural aspects of the code. These complementary masks introduce decoding diversity by providing multiple perspectives on bit-level dependencies within the same codeword. As a result, the ensemble design can compensate for decoding failures that may occur in any single masking configuration. Since all decoder modules operate in parallel, the system can achieve performance gains without incurring additional latency. Leveraging recent advances in AI model, especially the transformer architecture, we present the following novel contributions and results that align well with the requirements and challenges of 6G networks: 1. We propose a novel transformer-based decoder called CrossMPT, which shares key operational principles with conventional message-passing decoders. CrossMPT outperforms the original transformer-based decoder in both decoding performance and computational complexity, and has achieved state-of-the-art decoding performance among single neural decoders. 2. Building upon CrossMPT, we propose an AI-native foundation model for ECC decoding, FCrossMPT, which can be trained not only on a wide range of code lengths and code rates, but also on different code classes within a single model. We demonstrate that a single trained model can effectively decode diverse codes without requiring separate implementations or weight adjustments. This approach establishes a unified, code-agnostic decoder that serves as a true foundation model for ECC decoding. 3. To enhance the decoding performance of short-length codes, we propose CrossED, an ensemble decoder composed of multiple CrossMPT blocks in a parallel manner. Each CrossMPT block employs distinct PCMs as mask matrices, enabling the ensemble to capture diverse code structures. This architecture significantly improves decoding performance while maintaining decoding latency. Moreover, our CrossED is designed as a foundation model for ECC decoding, capable of decoding multiple code types without code-specific adaptations. I-B Related Work The diverse requirements of 6G networks have made a unified, code-agnostic decoding scheme a key objective [ 4 , 5 ] and this has led to strong interest in universal decoders. Classical algorithms like ordered statistics decoding (OSD) [ 13 ] and guessing random additive noise decoding (GRAND) [ 14 ] are considered universal as they can be applied to various code classes [ 5 ] . Similarly, our transformer-based decoder can function as a universal decoder; furthermore, it can be extended to become a foundation model–a single, pre-trained network that generalizes across diverse codes without requiring code-specific parameters or additional training. In addition, ensemble decoding is seen