Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs

Severin Bochem 1 , Victor J.B. Jung 2 , Arpan Suravi Prasad 2 , Francesco Conti 3 , Luca Benini 2 3 · 2024

Read Distributed Inference with Minimal Off-Chip Traffic for Transformers on Low-Power MCUs by Severin Bochem 1 , Victor J.B. Jung 2 , Arpan Suravi Prasad 2 , Francesco Conti 3 , Luca Benini 2 3 (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04372.txt.

Description

Severin Bochem and co-authors split transformer inference across tiny chips in a 2024 paper. Smart glasses and wearables want on-device language intelligence. Microcontrollers have too little memory for modern models. Shipping data off-chip burns the power budget fast.

Their method partitions inference across a team of microcontrollers. Weights stay stationary on-chip in the design. Off-chip traffic shrinks to a minimum as a result. A forty-two-million-parameter decoder ran across eight tiny chips. Energy per inference landed well under a millijoule. Latency fell to about half a millisecond per run. Speedup beat linear scaling against a single chip. A mobile transformer variant also ran distributed across four chips. Energy-delay product improved by an order of magnitude. The authors target next-generation interactive wearables. Distributed tiny hardware may host real contextual AI.

Catalog record

CREATORSeverin Bochem 1 , Victor J.B. Jung 2 , Arpan Suravi Prasad 2 , Francesco Conti 3 , Luca Benini 2 3
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04372

Browse the full library · About the Tower of Babel