Synthesizing Instruction-Tuning Datasets with Contrastive Decoding

Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki · 2026

Read Synthesizing Instruction-Tuning Datasets with Contrastive Decoding by Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-13538.txt.

Introduction

Ichinose and four colleagues untangle two things mixed together in answers from large language models: facts learned during pretraining and the habit of following instructions learned later. Their method, CoDIT, contrasts a tuned model against its untuned twin while generating answers, damping shared knowledge and boosting instruction-following behavior. Models trained on the resulting data beat models trained on plain generated answers.

Catalog record

CREATORTatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46618
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.13538

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Synthesizing Instruction-Tuning Datasets with Contrastive Decoding Tatsuya Ichinose, Youmi Ma, Masanari Oi, Ryuto Koike, Naoaki Okazaki Abstract Using responses generated by high-performing large language models (LLMs) for instruction tuning has become a widely adopted approach. However, the existing literature overlooks a property of LLM-generated responses: they conflate world knowledge acquired during pre-training with instruction-following capabilities acquired during post-training. We hypothesize that disentangling the instruction-following capabilities from pre-trained knowledge improves the effectiveness of instruction tuning. To this end, we propose CoDIT, a method that applies contrastive decoding between a post-trained model and its pre-trained counterpart during response generation. The method suppresses pre-trained knowledge shared between the two models while amplifying the instruction-following behavior acquired via post-training, resulting in responses that more purely reflect instruction-following capabilities. Experiment results demonstrate that models trained on datasets constructed via CoDIT consistently outperform those trained on directly generated responses. Training on our datasets also yields better performance than on existing publicly available instruction-tuning datasets across multiple benchmarks. Furthermore, we theoretically and empirically show that CoDIT can be interpreted as distilling the chat vector from parameter space to text space, enabling the transfer of instruction-tuning capabilities across models of different architectures. 1 1 1 The dataset and code are available at https://huggingface.co/datasets/Tatsuya-Ichinose/CoDIT and https://github.com/Tatsuya736482/contrastive_decoding_public , respectively. 1 Introduction Large language models (LLMs) have recently demonstrated strong capabilities in following user instructions. These capabilities are acquired through instruction tuning , a supervised fine-tuning process in which models learn to generate helpful responses to user instructions ( 47 ) . Instruction tuning requires instruction-response pairs as training data, but manually constructing such data is costly: instructions span diverse domains, necessitating domain experts to ensure response quality. To address this, existing work has demonstrated that using high-performing LLMs to generate responses can produce effective instruction-tuning datasets at scale ( 2 ; 48 ; 34 ; 33 ; 57 ; 59 ; 31 ) . However, directly using LLM-synthesized responses introduces a fundamental mismatch in the training objective. While the goal of instruction tuning is to teach models to follow instructions, LLM-generated responses conflate two distinct components, namely world knowledge acquired during pre-training (hereafter, pre-trained knowledge) and instruction-following abilities acquired during post-training. Previous work has shown that such pre-trained knowledge of a teacher cannot be effectively transferred to a student model via supervised fine-tuning ( 11 ) , suggesting that its presence in instruction-tuning data serves as noise rather than a helpful training signal. Therefore, we hypothesize that suppressing pre-training knowledge during response generation and preserving only instruction-following behavior can yield training data that more purely captures instruction-following ability, leading to more effective instruction tuning. Figure 1: Comparison of CoDIT with direct response generation. Existing methods use responses directly generated by the post-trained model (e.g., "In recent months, there are"). In contrast, CoDIT isolates the instruction-following behavior via contrastive decoding, which prioritizes tokens with high likelihood under the post-trained model and low likelihood under the pre-trained model. This work proposes CoDIT ( Co ntrastive D ecoding for I nstruction- T uning Dataset), a method that disentangles instruction-following abilities from pre-trained knowledge and suppresses the latter during response generation (Figure 1 ). To achieve this, we build on contrastive decoding ( 22 ) , a decoding strategy that improves output quality by favoring sequences with high likelihood under a large expert model over those under a smaller amateur model. We instantiate this framework using a post-trained model as the expert and its pre-trained checkpoint as the amateur. Such a formulation explicitly amplifies the difference between the two models, i.e., the instruction-following capabilities instilled by post-training, while suppressing the pre-trained knowledge they share. In experiments, we construct datasets using CoDIT with multiple open-weight LLMs – referred to as teacher models – and evaluate by training separate LLMs – referred to as student models – on the resulting data. As measured by WildBench ( 24 ) and AlpacaEval 2.0 ( 23 ) , models trained on CoDIT-generated responses consistently outperform those trained on responses directly generated by the teacher model across all nine teacher-student pairs, regardless of model scale or architecture. This validates our hypothesis that suppressing pre-trained knowledge during response generation yields more effective instruction-tuning data. Notably, our datasets also outperform existing instruction-tuning datasets ( 57 ; 49 ; 31 ; 17 ) , demonstrating their utility as training resources. Furthermore, we theoretically and empirically demonstrate that CoDIT can be interpreted as a text-space distillation of the chat vector ( 13 ) : the difference in model parameters before and after post-training. Chat vector operates in the parameter space, limiting its applicability to models with identical architectures. Our text-space distillation, however, lifts this constraint and enables the transfer of instruction-following capabilities across models of any scale or architecture. In short, the contributions of this work are as follows: (1) We propose CoDIT, a method for constructing instruction-tuning datasets that suppresses pre-trained knowledge during response generation, thereby isolating and emphasizing instruction-following capabilities. (2) We released datasets constructed via CoDIT that achieve state-of-the-art performance across multiple benchmarks. (3) We theoretically and empirically demonstrate that CoDIT can be interpreted as a text-space distillation of the chat vector, enabling the transfer of instruction-following capabilities across models of arbitrary architectures. 2 Methodology 2.1 CoDIT: Co ntrastive D ecoding for I nstruction- T uning Dataset In this study, we refer to a model that synthesizes responses as the teacher model , and a model trained on the synthesized responses as the student model 2 2 2 We treat post-trained models as expert models, noting that these models may have undergone additional post-training stages beyond instruction tuning, such as reinforcement learning. An ablation study on the effect of different post-training stages is provided in Section 4.2 and Appendix E . . CoDIT applies contrastive decoding ( 22 ) to the teacher model during response generation, leveraging the pre-trained model as the amateur and the post-trained model as the expert. Specifically, given an instruction x , the goal is to generate a response y . Let \theta_{\text{post}} and \theta_{\text{pre}} denote the parameters of the post-trained and the pre-trained version of the teacher model, respectively. The response y is decoded token by token, where each token is selected to maximize the difference between the log-probabilities assigned the post-trained and pre-trained model. For example, the i -th token y_{i} is selected as: \displaystyle s(v;x,y_{<i}) \displaystyle=\log P(v\mid x,y_{<i};\theta_{\text{post}})-\log P(v\mid x,y_{<i};\theta_{\text{pre}}), (1) \displaystyle y_{i} \displaystyle=\arg\max_{v\in\mathcal{V}}s(v;x,y_{<i}), (2) where s(v;x,y_{<i}) is the contrastive score, v is a candidate token chosen from the vocabulary \mathcal{V} , and y_{<i} is the sequence of previously decoded tokens. The full response y is obtained by repeating this selection autoregressively until a stop token is reached. However, as noted in prior work ( 22 ) , unrestricted contrastive decoding using ( 2 ) can produce degraded outputs: tokens with very low probability under the pre-trained model may be overemphasized, while tokens with high probabilities under both models may be underemphasized. This can suppress contextually appropriate tokens, degrading the fluency of the generated text. To mitigate this, we follow 22 and apply a plausibility constraint, which restricts candidate tokens to those assigned sufficiently high probability by the post-trained model. Formally, let \alpha\in[0,1] be a hyperparameter controlling the strictness of the constraint. We define

Read the complete text