SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations
Gaurav Sarkar, Syed Affan Daimi, Jay Gala, Subarna Tripathi · 2025
Read SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations by Gaurav Sarkar, Syed Affan Daimi, Jay Gala, Subarna Tripathi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23942.txt.
Description
Gaurav Sarkar, Syed Affan Daimi, Jay Gala, and Subarna Tripathi blend two gate curves for neural nets in a 2025 paper. Gate curves shape how each net layer fires, and top picks each suit one field best. No past pick can bend its shape to fit each layer's needs.
Their blend lets each layer pick its own mix of the two curves. A fixed kin of one curve, with two small knobs, forms one side of the blend. The blend costs just three extra knobs per layer, the team notes. On BERT-style IMDB tests over five seeds, it matched peak accuracy of 81.31% and cut seed-to-seed variance by 42% compared with GELU. On WikiText-103, it gave the lowest validation perplexity, 49.10, of all baselines tried.
Catalog record
| CREATOR | Gaurav Sarkar, Syed Affan Daimi, Jay Gala, Subarna Tripathi |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-23942