ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks
Dhruv Saini, Rohan Pandey · 2026
Read ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks by Dhruv Saini, Rohan Pandey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01080.txt.
Description
Saini and Pandey move the gains of slow thinking into a fast model's weights in this 2026 paper. Thinking models solve hard tasks by writing a reasoning trace first. That trace costs time, tokens, and serving complexity. Their procedure, ThinkSwitch, co-trains a paired instruct and thinking checkpoint in a loop.
Each round, the thinking model answers task prompts. The trace is stripped, and the answer only pairs are distilled into the instruct model with QLoRA. Spherical weight interpolation then rebuilds a fresh thinking model. Only the prompts are human supplied. On 30 AIME questions, the instruct model climbs from 10 to 20 correct, they report. The thinking model rises from 14 to 22. A PubMedQA subset shows like gains. The whole run uses 15 prompts per domain and costs under three dollars on one cloud card.
Catalog record
| CREATOR | Dhruv Saini, Rohan Pandey |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 21 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2606.01080