NITRO: LLM Inference on Intel ® Laptop NPUs

Anthony Fei, Mohamed S. Abdelfattah · 2024

Read NITRO: LLM Inference on Intel ® Laptop NPUs by Anthony Fei, Mohamed S. Abdelfattah (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11053.txt.

Description

Anthony Fei and Mohamed Abdelfattah build a framework that lets large language models run on the AI chip inside Intel laptops. Their 2024 paper starts from a gap: the chip, a neural processing unit, ships in the Core Ultra line, but the maker's software only runs fixed models on it. Chat models grow their text one token at a time, which is anything but fixed.

Their kit, NITRO, sits on the maker's toolkit and reshapes the transformer so the chip can serve it. The paper walks through the changes in detail, gives speed benchmarks, and lists the next steps for the package. The code is public, so laptop builders can test chat on the low-power chip rather than the main cores.

Catalog record

CREATORAnthony Fei, Mohamed S. Abdelfattah
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE28 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-11053

Browse the full library · About the Tower of Babel