A 35B Hybrid-Attention Mixture-of-Experts Model on a 6 GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi ( sm_20 )

A. C. Opus, J. Q. Lu · 2026

Read A 35B Hybrid-Attention Mixture-of-Experts Model on a 6 GB 2011 GPU: Hand-Written 4-bit CUDA Inference for Fermi ( sm_20 ) by A. C. Opus, J. Q. Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-24031.txt.

Description

Opus and Lu run a thirty five billion parameter language model on a graphics card from 2011. The card has six gigabytes of memory. They hand wrote the entire inference engine to do it, in a 2026 paper on computer systems. The model occupies about ten and a half gigabytes even at four bit precision, twice the card's memory. Modern software cannot help: the card predates tensor cores and every current compiler toolchain. The authors split the work: expert weights stream from main memory for GPU prefill, while text generation runs on the CPU through a hand built integer kernel.

Tuning cuts prefill on a long prompt from fifty seven seconds to thirty seven and a half, and triples generation speed. A snapshot cache restores prefix reuse. It collapses a repeated seventy eight second prefill to half a second. Several plausible optimizations fail, mapping the old hardware's floor.

Catalog record

CREATORA. C. Opus, J. Q. Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.24031

Browse the full library · About the Tower of Babel