ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks

Dhruv Saini, Rohan Pandey · 2026

Read ThinkSwitch: Context Distillation with LoRA and Weight Interpolation for Specific-Purpose Reasoning Tasks by Dhruv Saini, Rohan Pandey (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-01080.txt.

Description

Saini and Pandey move the gains of slow thinking into a fast model's weights in this 2026 paper. Thinking models solve hard tasks by writing a reasoning trace first. That trace costs time, tokens, and serving complexity. Their procedure, ThinkSwitch, co-trains a paired instruct and thinking checkpoint in a loop.

Each round, the thinking model answers task prompts. The trace is stripped, and the answer only pairs are distilled into the instruct model with QLoRA. Spherical weight interpolation then rebuilds a fresh thinking model. Only the prompts are human supplied. On 30 AIME questions, the instruct model climbs from 10 to 20 correct, they report. The thinking model rises from 14 to 22. A PubMedQA subset shows like gains. The whole run uses 15 prompts per domain and costs under three dollars on one cloud card.

Catalog record

CREATORDhruv Saini, Rohan Pandey
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE21 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.01080

Browse the full library · About the Tower of Babel