NITRO: LLM Inference on Intel ® Laptop NPUs
Anthony Fei, Mohamed S. Abdelfattah · 2024
Read NITRO: LLM Inference on Intel ® Laptop NPUs by Anthony Fei, Mohamed S. Abdelfattah (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11053.txt.
Description
Anthony Fei and Mohamed Abdelfattah build a framework that lets large language models run on the AI chip inside Intel laptops. Their 2024 paper starts from a gap: the chip, a neural processing unit, ships in the Core Ultra line, but the maker's software only runs fixed models on it. Chat models grow their text one token at a time, which is anything but fixed.
Their kit, NITRO, sits on the maker's toolkit and reshapes the transformer so the chip can serve it. The paper walks through the changes in detail, gives speed benchmarks, and lists the next steps for the package. The code is public, so laptop builders can test chat on the low-power chip rather than the main cores.
Catalog record
| CREATOR | Anthony Fei, Mohamed S. Abdelfattah |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 28 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11053