Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition

Tyler McDonald, Ali Emami · 2025

Read Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition by Tyler McDonald, Ali Emami (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20946.txt.

Introduction

Tyler McDonald and Ali Emami present, in a 2025 paper, a prompting trick that lets small language models borrow the reasoning of big ones without costly retraining. Classic distillation, where a student model copies a teacher, usually needs heavy fine-tuning. Their method works through the prompt alone, by breaking each problem into a trace of smaller questions.

Called Trace-of-Thought Prompting, the setup has the teacher's reasoning style guide the student step by step. The split makes the logic easier to inspect and lets a human step in mid-way. On two maths sets, students gained up to 113 percent in accuracy on the easier set and twenty-one percent on the harder one, with the smallest models gaining most. The authors see a path to open models that can later teach others.

Catalog record

CREATORTyler McDonald, Ali Emami
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20946

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Trace-of-Thought Prompting: Investigating Prompt-Based Knowledge Distillation Through Question Decomposition Tyler McDonald, Ali Emami Abstract Knowledge distillation allows smaller neural networks to emulate the performance of larger, teacher models with reduced computational demands. Traditional methods for Large Language Models (LLMs) often necessitate extensive fine-tuning, which limits their accessibility. To address this, we introduce Trace-of-Thought Prompting, a novel framework designed to distill critical reasoning capabilities from high-resource teacher models (over 8 billion parameters) to low-resource student models (up to 8 billion parameters). This approach leverages problem decomposition to enhance interpretability and facilitate human-in-the-loop interventions. Empirical evaluations on the GSM8K and MATH datasets show that student models achieve accuracy gains of up to 113% on GSM8K and 21% on MATH, with significant improvements particularly notable in smaller models like Llama 2 and Zephyr. Our results suggest a promising pathway for open-source, low-resource models to eventually serve both as both students and teachers, potentially reducing our reliance on high-resource, proprietary models. Figure 1: A Visual Depiction of our Trace-of-Thought prompting strategy on a GSM8K problem instance ( Cobbe et al., 2021 ) . 1 Introduction Knowledge distillation, as initially proposed by Hinton et al. (2015) , involves leveraging the outputs of larger neural networks as soft targets to train smaller, more efficient networks. This method, primarily applied to tasks like MNIST LeCun et al. (1998) in computer vision, uses computationally heavy teacher models to facilitate equivalent reasoning capacities in smaller models, substantially reducing computational demands on the user. As the popularity of Large Language Models (LLMs) has surged, adaptations of this technique have been explored, particularly through fine-tuning based on the outputs of these large models. However, these adaptations often introduce a significant computational overhead and necessitate a deep understanding of machine learning, limiting their accessibility for average consumers ( Xu et al., 2024 ; Gu et al., 2024 ; Liu et al., 2024 ; Zhong et al., 2024 ) . Concurrently, the rapid development of LLMs has been paralleled by innovations in prompt engineering—the strategic design of prompts to enhance reasoning and explore various problem-solving pathways ( Sahoo et al., 2024 ; Chen et al., 2024a ) . Methods such as Chain-of-Thought Prompting and Self-Consistency have demonstrated the potential of LLMs to engage in complex reasoning and provide novel solutions to challenging problems ( Wei et al., 2023 ; Wang et al., 2023b ; Yao et al., 2023 ; Wang et al., 2023a ) . Nevertheless, these approaches typically operate within a single contextual framework and rely heavily on the innate reasoning capabilities of models, often failing when applied to smaller, open-source variants ( Touvron et al., 2023 ; Tunstall et al., 2023 ; Xu et al., 2023 ) . This suggests a critical need for a more adaptable and scalable approach to knowledge distillation that can leverage the advances in prompt engineering for broader accessibility and effectiveness. In response to this need, we explore the intersection of prompt engineering and knowledge distillation through a novel concept we term prompt-based knowledge distillation . This approach utilizes in-context learning (ICL) to emulate traditional distillation processes within the accessible framework of LLM prompting, mirroring the cognitive process of a student learning from a teacher ( Brown et al., 2020 ) . To implement this concept, we introduce Trace-of-Thought Prompting , a technique that decomposes complex arithmetic reasoning problems into manageable steps, facilitating the distillation of critical reasoning skills from high-resource models to their low-resource counterparts (see Figure 1 ). This strategy not only improves the performance of low-resource models but also demonstrates their potential to serve as effective teachers themselves. Our contributions to this novel extension of knowledge distillation are threefold: 1. We propose Trace-of-Thought Prompting , a novel framework for prompt-based knowledge distillation. This approach allows knowledge transfer from high-resource models (greater than 8 billion parameters) to low-resource models (up to 8 billion parameters) through structured problem decomposition. 2. We demonstrate significant performance enhancements across two complex arithmetic reasoning datasets. By applying Trace-of-Thought Prompting, we improve the performance of low-resource models on the GSM8K dataset by 113% and on the MATH dataset by XYZ. Our results also illustrate the effectiveness of low-resource models, like Llama 2 and Zephyr, in achieving performance gains that make them viable alternatives to their high-resource counterparts. 3. Our extended analyses demonstrate that Trace-of-Thought Prompting not only enhances quantitative performance metrics but also improves the transparency of the problem-solving process. This transparency allows for more effective human-in-the-loop interventions, where incorrect or suboptimal reasoning paths generated by the models can be identified and corrected before execution. 2 Related Work Decomposed reasoning. Traditional question decomposition methods, including Plan & Solve Prompting and Progressive Hint Prompting, engage in single-context question decomposition, integrating a planning stage followed by an execution phase ( Wang et al., 2023a ; Press et al., 2023 ; Sun et al., 2023 ) . More sophisticated recursive techniques, such as Least-to-Most Prompting, sequentially append results to enhance the context for subsequent prompts ( Zhou et al., 2023 ; Dua et al., 2022 ; Khot et al., 2023 ; Zheng et al., 2023 ) . These methodologies, however, face significant challenges: single-context systems fail to effectively leverage multiple models simultaneously, limiting flexibility and adaptability; recursive techniques, while intricate, hold the potential to lead to extended input sequences and excessive computational demands by virtue of their repetitive nature ( Guo et al., 2024 ; Mohtashami et al., 2024 ; Juneja et al., 2024 ) . Our Trace-of-Thought Prompting addresses these issues by facilitating dynamic, multi-model cooperation without the need for expansive input chains, streamlining the reasoning process across varied contexts. Open-source language modeling. The rise of open-source models like WizardLM, Zephyr, and Llama has democratized access to language model customization and deployment ( Xu et al., 2023 ; Touvron et al., 2023 ; Tunstall et al., 2023 ; Gunasekar et al., 2023 ; Team et al., 2024 ) . Despite their accessibility, the teams behind these models report frequent deficiencies in complex reasoning tasks in small variants, underscoring a persistent correlation between model size and reasoning capabilities ( Agrawal et al., 2024 ; Chen et al., 2024c ; Zhang et al., 2024 ) . Trace-of-Thought Prompting enhances these models’ performance by distilling complex reasoning from larger models into manageable steps, effectively bridging the gap in reasoning prowess without extensive hardware demands. Tandem and Socratic reasoning. The exploration of collaborative problem-solving in model suites, such as Socratic Chain-of-Thought and Socratic Questioning, introduces novel ways to utilize multiple models in a cohesive manner ( Shridhar et al., 2023 ; Qi et al., 2023 ; Chang, 2024 ; Zeng et al., 2022 ; Goel et al., 2024 ) . However, these approaches encounter difficulties with managing large context sizes and reliance on fine-tuning ( Li et al., 2024 ; Wang et al., 2024 ) . Our work contributes to this area by implementing a structured approach that minimizes token bloat and fine-tuning dependency, offering a more efficient and scalable solution for collaborative reasoning within LLM environments. Figure 2: Visual depiction of the methods employed during experimentation. Trace-of-Thought provides a novel decomposition framework in a linear manner. Prompt Type Template Standard “<question>.” Chain-of-Thought “<question>. Think step-by-step. ” Trace-of-Thought - Delegation “ Create very short step-by-step prompts

Read the complete text