TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference
2026
Read TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-05145.txt.
Description
Speculative decoding speeds up language model serving by drafting tokens with a small model and checking them with the big one. Jiyoung Park and 3 co-authors present TIDE in this 2026 systems paper, accepted at the SC'26 supercomputing conference. The practical problem is drift: workloads change, and a draft model tuned for yesterday's traffic can slow today's serving down.
TIDE adapts the draft model online, inside the serving engine itself. It recycles the target model's own hidden states, produced anyway during inference, as free training signals, and switches speculation and training on only when they pay off. On heterogeneous clusters it places inference and training on the GPU classes that suit each. Across real workloads, throughput rose up to about 1.66 times over no speculation, with training time cut by up to about three times and storage by a factor of 24 against existing draft training approaches.
Catalog record
| YEAR | 2026 |
|---|---|
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2602-05145