PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

Daniel Nobrega Medeiros · 2026

Read PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks by Daniel Nobrega Medeiros (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13347.txt.

Description

Daniel Nobrega Medeiros introduces PolyGLU, a new activation design for transformer feed-forward networks. Brains use many neurotransmitters for distinct signal modes, while transformers use one fixed activation for all neurons. PolyGLU lets each neuron route among four activation functions by learned preference plus input-based gating. He trains PolychromaticLM, a 597M-parameter model, on about 10B tokens with one GPU. Routing settled into near-fixed choices with no sparsity loss, and early layers favored GELU while deep layers favored Tanh.

Catalog record

CREATORDaniel Nobrega Medeiros
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33216
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13347

Browse the full library · About the Tower of Babel