DAPA: D istribution A ware P iecewise A ctivation Functions for On-Device Transformer Inference and Training
Maoyang Xiang, Bo Wang · 2026
Read DAPA: D istribution A ware P iecewise A ctivation Functions for On-Device Transformer Inference and Training by Maoyang Xiang, Bo Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-19338.txt.
Description
Xiang and Wang design a cheaper activation function for transformers that run on small devices. Their DAPA bends its shape to fit where input values actually cluster. Segments are finer in dense zones and coarser in sparse ones. A weighted error rule then quantizes the shape for hardware use. On chip design tests, GELU math runs 16 times faster with 16 times fewer DSP blocks. Scores on vision transformers and GPT-2 stay level or improve. The design targets both test-time use and on-device training.
Catalog record
| CREATOR | Maoyang Xiang, Bo Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 29427 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.19338