LLM-42 : Enabling Determinism in LLM Inference with Verified Speculation

Raja Gond, Aditya K Kamath, Ramachandran Ramjee, and Ashish Panwar, [1ex] Microsoft Research · 2026

Read LLM-42 : Enabling Determinism in LLM Inference with Verified Speculation by Raja Gond, Aditya K Kamath, Ramachandran Ramjee, and Ashish Panwar, [1ex] Microsoft Research (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-17768.txt.

Description

Raja Gond, Aditya K Kamath, Ramachandran Ramjee, and Ashish Panwar of Microsoft Research study a quirk of large language models. The same prompt can give a different reply each time the model runs. Their 2026 preprint traces the cause to math inside the machine. Adding numbers in a changed order can change the final sum, and batch sizes shift that order.

One fix is to turn off dynamic batching, but that slows the system down a lot. The team instead works on a way to keep results stable while the speed stays high. Their title calls the method verified speculation. Steady output matters for tests, audits, and any use where the same input must give the same output.

Catalog record

CREATORRaja Gond, Aditya K Kamath, Ramachandran Ramjee, and Ashish Panwar, [1ex] Microsoft Research
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-17768

Browse the full library · About the Tower of Babel