Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs
Severin Bochem 1 , Victor J.B. Jung 2 , Arpan Suravi Prasad 2 , Francesco Conti 3 , Luca Benini 2 3 · 2024
Read Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs by Severin Bochem 1 , Victor J.B. Jung 2 , Arpan Suravi Prasad 2 , Francesco Conti 3 , Luca Benini 2 3 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04372.txt.
Description
Severin Bochem and co-authors split transformer inference across tiny chips in a 2024 paper. Smart glasses and wearables want on-device language intelligence. Microcontrollers have too little memory for modern models. Shipping data off-chip burns the power budget fast.
Their method partitions inference across a team of microcontrollers. Weights stay stationary on-chip in the design. Off-chip traffic shrinks to a minimum as a result. A forty-two-million-parameter decoder ran across eight tiny chips. Energy per inference landed well under a millijoule. Latency fell to about half a millisecond per run. Speedup beat linear scaling against a single chip. A mobile transformer variant also ran distributed across four chips. Energy-delay product improved by an order of magnitude. The authors target next-generation interactive wearables. Distributed tiny hardware may host real contextual AI.
Catalog record
| CREATOR | Severin Bochem 1 , Victor J.B. Jung 2 , Arpan Suravi Prasad 2 , Francesco Conti 3 , Luca Benini 2 3 |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-04372