Collapse-Free Prototype Readout Layer for Transformer Encoders

Giansalvo Cirrincione, Rahul Ranjeev Kumar · 2026

Read Collapse-Free Prototype Readout Layer for Transformer Encoders by Giansalvo Cirrincione, Rahul Ranjeev Kumar (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-03850.txt.

Description

Giansalvo Cirrincione and Rahul Ranjeev Kumar built DDCL-Attention, a prototype readout layer that replaces mean pooling and class tokens in transformer encoders. A small set of global prototypes gathers tokens by soft matching, yielding compact summaries at linear cost. An exact loss split into a rebuild term and a spread term stops prototypes from collapsing into copies. Tests on four data sets backed the theory, with full codebook use beating hard vector quantization.

Catalog record

CREATORGiansalvo Cirrincione, Rahul Ranjeev Kumar
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE78614
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.03850

Browse the full library · About the Tower of Babel