INCRT: An Incremental Transformer That Determines Its Own Architecture

Giansalvo Cirrincione · 2026

Read INCRT: An Incremental Transformer That Determines Its Own Architecture by Giansalvo Cirrincione (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-10703.txt.

Description

Cirrincione introduces INCRT, a transformer that settles its own architecture during training. Starting from one head, it adds heads when needed and prunes redundant ones. Each growth choice follows one geometric quantity from the task's structure, with no hand-tuned schedule. Two theorems back the design: convergence to a minimal sufficient setup and a cap on head count. Tests on SARS-CoV-2 classification and SST-2 sentiment match predictions within 12 percent. Final models match or beat BERT-base with three to seven times fewer parameters and no pretraining.

Catalog record

CREATORGiansalvo Cirrincione
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63223
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.10703

Browse the full library · About the Tower of Babel