A 35B Hybrid-Attention Mixture-of-Experts Model on a 6 GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi ( sm_20 )
A. C. Opus, J. Q. Lu · 2026
Read A 35B Hybrid-Attention Mixture-of-Experts Model on a 6 GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi ( sm_20 ) by A. C. Opus, J. Q. Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24031.txt.
Description
Opus and Lu run a thirty five billion parameter language model on a graphics card from 2011. The card has six gigabytes of memory. They hand wrote the entire inference engine to do it, in a 2026 paper on computer systems. The model occupies about ten and a half gigabytes even at four bit precision, twice the card's memory. Modern software cannot help: the card predates tensor cores and every current compiler toolchain. The authors split the work: expert weights stream from main memory for GPU prefill, while text generation runs on the CPU through a hand built integer kernel.
Tuning cuts prefill on a long prompt from fifty seven seconds to thirty seven and a half, and triples generation speed. A snapshot cache restores prefix reuse. It collapses a repeated seventy eight second prefill to half a second. Several plausible optimizations fail, mapping the old hardware's floor.
Catalog record
| CREATOR | A. C. Opus, J. Q. Lu |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.24031