TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference

2026

Read TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05145.txt.

Description

Speculative decoding speeds up language model serving by drafting tokens with a small model and checking them with the big one. Jiyoung Park and 3 co-authors present TIDE in this 2026 systems paper, accepted at the SC'26 supercomputing conference. The practical problem is drift: workloads change, and a draft model tuned for yesterday's traffic can slow today's serving down.

TIDE adapts the draft model online, inside the serving engine itself. It recycles the target model's own hidden states, produced anyway during inference, as free training signals, and switches speculation and training on only when they pay off. On heterogeneous clusters it places inference and training on the GPU classes that suit each. Across real workloads, throughput rose up to about 1.66 times over no speculation, with training time cut by up to about three times and storage by a factor of 24 against existing draft training approaches.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-05145

Browse the full library · About the Tower of Babel