Smaller Language Models Are Better Instruction Evolvers
Tingfeng Hui, Lulu Zhao, Guanting Dong, Yaqi Zhang, Hua Zhou, Sen Su · 2024
Read Smaller Language Models Are Better Instruction Evolvers by Tingfeng Hui, Lulu Zhao, Guanting Dong, Yaqi Zhang, Hua Zhou, Sen Su (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-11231.txt.
Introduction
Tingfeng Hui and five co-authors argue that small models write better training instructions than big ones in a 2024 paper. Instruction tuning teaches models to follow commands, and rich, varied instructions matter. The field has assumed giant models should generate them, so most pipelines use the biggest available.
Across three evolution scenarios, the authors found the opposite: smaller models produced more effective instructions. Their outputs ranged more widely, yielding more complex and varied training data. Existing quality scores missed this, so the authors propose a complexity-aware metric that judged the data better. The result questions a costly default in model training, and the team releases its code for others to check.
Catalog record
| CREATOR | Tingfeng Hui, Lulu Zhao, Guanting Dong, Yaqi Zhang, Hua Zhou, Sen Su |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-11231
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Smaller Language Models Are Better Instruction Evolvers Tingfeng Hui, Lulu Zhao, Guanting Dong, Yaqi Zhang, Hua Zhou, Sen Su Abstract Instruction tuning has been widely used to unleash the complete potential of large language models. Notably, complex and diverse instructions are of significant importance as they can effectively align models with various downstream tasks. However, current approaches to constructing large-scale instructions predominantly favour powerful models such as GPT-4 or those with over 70 billion parameters, under the empirical presumption that such larger language models (LLMs) inherently possess enhanced capabilities. In this study, we question this prevalent assumption and conduct an in-depth exploration into the potential of smaller language models (SLMs) in the context of instruction evolution. Extensive experiments across three scenarios of instruction evolution reveal that smaller language models (SLMs) can synthesize more effective instructions than LLMs. Further analysis demonstrates that SLMs possess a broader output space during instruction evolution, resulting in more complex and diverse variants. We also observe that the existing metrics fail to focus on the impact of the instructions. Thus, we propose Instruction Complex-Aware IFD (IC-IFD), which introduces instruction complexity in the original IFD score to evaluate the effectiveness of instruction data more accurately. Our source code is available at: https://github.com/HypherX/Evolution-Analysis 1 Introduction Large Language Models (LLMs) have demonstrated exceptional performance in various NLP tasks and are widely integrated into a variety of applications, represented by ChatGPT and Copilot ( Ouyang et al. 2022 ; OpenAI 2023 ; Dubey et al. 2024 ) . A key factor in unleashing the full potential of these models is high-quality instruction tuning data, which plays a crucial role in post-training and enhances their effectiveness as AI assistants. In particular, incorporating more complex and diverse instructions allows models to better align with different domains and tasks, boosting their performance in a variety of downstream applications ( Zhang et al. 2023 ) . However, generating such diverse instructions remains time-consuming and labor intensive ( Zheng et al. 2024a ; Zhao et al. 2024 ; Liu et al. 2024 ) , which undoubtedly presents a significant challenge for the automated and scalable alignment of LLMs. Recently, a series of efforts utilizing LLMs for automatic instruction evolution have garnered sustained attention from the community. Specifically, foundational work like Self-Instruct ( Wang et al. 2023 ) begins with a small set of seed instructions and uses a powerful supervision model to obtain a large number of synthetic instructions. Furthermore, Evol-Instruct ( Xu et al. 2024a ) refines and evolves existing instructions to produce more complex variants. However, previous studies mainly favour strong LLMs like GPT-4 or those with more than 70 billion parameters to synthesize instructions, empirically assuming that larger language models inherently have superior instruction evolution capabilities. But is this really the case? Recently, Xu et al. 2024c propose the Larger Models’ Paradox, which points out that larger models do not necessarily lead to better performance when generating responses, but it overlooks the analysis of instructions. We propose that smaller language models, which require less computational demand and have lower instruction following capabilities, may provide a more efficient and effective alternative for evolving more complex and diverse instructions. To gain insight into this, we investigate the differences between smaller language models (SLMs) and larger language models (LLMs) in generating high-quality instructions. Specifically, given a set of base models and seed instructions, we are particularly interested in the following research question: RQ1: Do SLMs Perform Better than LLMs in Evolving Instructions? In response to this, we conduct comprehensive experiments across three distinct instruction evolution scenarios: Evol-Instruct, AutoIF ( Dong et al. 2024 ) , and Auto Evol-Instruct ( Zeng et al. 2024 ) . In these experiments, we use small ( \sim 8B) and large ( \sim 70B) models from the Llama-3.1 and Qwen-2 families to evolve and synthesize new instructions, while also fine-tuning various backbone models. The experimental results across all three scenarios consistently indicate that larger, more powerful LLMs do not outperform SLMs in evolving effective instructions. More interestingly, SLMs even demonstrate the capability to evolve more complex and diverse instructions. To further investigate why more powerful LLMs perform worse than SLMs in generating new instructions, we subsequently pose the second research question. RQ2: Why do SLMs Outerperform LLMs in Evolving Instructions? To better understand why more powerful LLMs underperform compared to SLMs in evolving instructions, we compare the top-1 token probabilities of both models during the synthetic of instructions. Our findings demonstrate that LLMs, due to their superior instruction following capabilities, tend to generate a higher proportion of high-probability top-1 tokens when evolving new instructions. This overconfidence in token generation results in a narrower output space. In contrast, SLMs can generate a wider variety of tokens, leading to more complex and diverse instructions. To further investigate what kind of instruction data is effective, we propose the third research question. RQ3: How Do We Determine Whether An Instruction is Effective without Instruction Tuning? Evaluations that do not require instruction tuning can more efficiently assess instruction data. Recent such evaluations often fail to account for the impact of the instructions themselves. For instance, reward models ( Cai et al. 2024 ) are commonly used to assess the quality of responses generated based on a given instruction, yet they tend to overlook the quality of the instruction itself. Similarly, while the IFD score ( Li et al. 2024 ) measures the influence of instructions on response generation, it neglects the effect of the instruction’s inherent complexity. We introduce the Instruction Complex-Aware IFD (IC-IFD) score, which incorporates the difficulty of the instruction as a penalty term in the original IFD. We conduct extensive filtering instruction data experiments, and the results demonstrate that the IC-IFD score provides a more accurate assessment of instruction data, particularly in scenarios where the instructions exhibit higher complexity levels. In summary, our key contributions are as follows: (1) To the best of our knowledge, we are the first to comprehensively explore the performance discrepancies between SLMs and LLMs in synthesizing instructions. (2) Extensive experimental results demonstrate that SLMs have a broader output space, leading to evolving more complex and diverse instructions. (3) We propose the IC-IFD score, which introduces the difficulty of the instruction as a penalty term. Comprehensive experiments show that IC-IFD can more accurately assess the effectiveness of instruction data without instruction tuning. Model Instruction Following (IFEval) Math Reasoning Code Generation Pr.(S) In.(S) Pr.(L) In.(L) GSM8K MATH HumanEval MBPP Supervised Model: Llama-3.1-70B-Instruct Mistral-7B-v0.3 19.59 31.77 22.74 34.65 33.89 3.16 24.39 6.00 DeepSeek-7B 36.23 48.20 41.04 52.52 48.07 2.96 28.66 33.00 Llama-3.2-3B 40.11 50.84 43.81 54.43 53.75 6.60 35.98 36.00 Llama-3-8B 33.83 46.28 36.41 49.28 63.00 7.62 43.90 36.20 Llama-3.1-8B 34.57 46.04 38.81 50.48 64.22 11.32 51.22 40.60 InternLM-2-7B 40.85 53.48 44.54 56.95 68.31 19.50 56.10 40.40 Supervised Model: Llama-3.1-8B-Instruct Mistral-7B-v0.3 24.40 35.01 26.25 37.53 40.18 2.84 29.27 19.60 DeepSeek-7B 36.60 48.08 41.77 53.12 47.92 3.56 34.76 33.80 Llama-3.2-3B 41.59 53.48 45.66 57.07 55.12 7.32 39.02 32.80 Llama-3-8B 35.49 47.00 39.56 50.72 63.38 11.44 48.17 37.60 Llama-3.1-8B 38.45 50.96 43.81 55.28 67.10 13.12 48.78 41.60 InternLM-2-7B 43.07 54.80 47.32 58.39 68.08 20.32 57.93 40.80 Table 1: Comparison of performance with Llama-3.1-8B-Instruct and Llama-3.1-70B-Instruct as supervised models under Evol-Instruct scenario. 2