One-Pass to Reason: Token Duplication and Block-Sparse Mask for Efficient Fine-Tuning on Multi-Turn Reasoning

Ritesh Goru, Shanay Mehta, Prateek Jain · 2025

Read One-Pass to Reason: Token Duplication and Block-Sparse Mask for Efficient Fine-Tuning on Multi-Turn Reasoning by Ritesh Goru, Shanay Mehta, Prateek Jain (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18246.txt.

Introduction

Goru, Mehta, and Jain found a way to fine-tune language models on multi-turn reasoning data in a single pass. Normally each turn of a conversation needs its own forward pass, because reasoning tokens from one turn must stay hidden in later turns. That makes training cost grow steeply with the number of turns. Their trick duplicates the response tokens and uses a custom block-sparse attention mask to enforce the same hiding rules inside one pass.

They prove the method gives losses identical to the multi-pass approach while cutting the time growth from cubic to quadratic in the number of turns, with no extra memory cost. In practice this means much faster fine-tuning with unchanged accuracy. The authors release their implementation openly for others to use.

Catalog record

CREATORRitesh Goru, Shanay Mehta, Prateek Jain
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18246

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

One-Pass to Reason: Token Duplication and Block-Sparse Mask for Efficient Fine-Tuning on Multi-Turn Reasoning Ritesh Goru, Shanay Mehta, Prateek Jain Abstract Fine-tuning LLM on multi-turn reasoning datasets requires N (number of turns) separate forward passes per conversation due to reasoning token visibility constraints, as reasoning tokens for a turn are discarded in subsequent turns. We propose duplicating response tokens along with a custom attention mask to enable single-pass processing of entire conversations. We prove our method produces identical losses to the N-pass approach while reducing time complexity from O ( N 3 ) O\bigl(N^{3}\bigl) to O ( N 2 ) O\bigl(N^{2}\bigl) and maintaining the same memory complexity for a transformer based model. Our approach achieves significant training speedup while preserving accuracy. Our implementation is available online 2 2 2 https://github.com/devrev/One-Pass-to-Reason . LLM Large Language Model GRPO Group Relative Policy Optimisation 1 Introduction Recent progress in LLM has sparked a shift from models that directly generate final responses to those that perform explicit intermediate reasoning before generating responses (referred to as reasoning models). Open-source reasoning models, such as DeepSeek-R1 ( Guo et al., 2025 ) , demonstrate high performance on several benchmarks. However, these existing reasoning models were trained primarily on single-turn reasoning data. While numerous studies have investigated fine-tuning LLMs for multi-turn dialogues to improve coherence, context awareness, tool-calling ( Wang et al., 2025 ; Rebedea et al., 2024 ) , these approaches assume non-reasoning dialogues. Training LLMs for multi-turn reasoning conversations presents novel challenges in managing token visibility. Following industry-standard practices for multi-turn conversations ( OpenAI, 2024 ; Anthropic, 2025 ) , reasoning models generate internal reasoning tokens, produce a response, and then discard the reasoning tokens from the context in subsequent turns. This creates two fundamental constraints that cannot be addressed with standard multi-turn optimization techniques: (1) Visibility Constraints : Reasoning tokens must be visible during generation but hidden from subsequent conversation turns, requiring conditional visibility that static attention masks cannot satisfy. (2) Position ID Discrepancy : Response tokens follow reasoning tokens during generation but directly follow human messages in a later context, creating positional misalignment. While prior works have explored masking techniques and position ID assignments to control information flow and enable selective attention within sequences for various pre-training objectives or efficiency gains ( Wang & Hegde, 2024 ; Du et al., 2022 ; Raffel et al., 2020 ) , none address the specific challenges of multi-turn reasoning conversations where reasoning tokens must be conditionally visible across turns. This paper addresses these challenges with two primary contributions. (1) We present a theoretical framework featuring a block-sparse visibility mask and strategic position ID assignment scheme that enables processing an entire multi-turn reasoning conversation in a single forward pass while maintaining training correctness (Theorem 2.1 ). (2) Due to the absence of a publicly available multi-turn reasoning dataset (to the best of our knowledge), we create and release a novel dataset, \text{MathChat}_{\text{sync}}\text{Reasoning} , in which each assistant message is augmented with synthetically generated reasoning. (3) We provide comprehensive empirical validation for the proposed framework on Qwen3 models. Notation. We use \mathcal{D} to denote a multi-turn reasoning dataset where each conversation c\in\mathcal{D} consists of alternating human messages h_{i} and assistant messages a_{i} such that c=(h_{i},a_{i})_{i=1}^{N} for N turns. Each assistant message a_{i} comprises thinking tokens t_{i} and response tokens r_{i} . We denote \mathcal{H}_{<i}=(h_{j},r_{j})_{j=1}^{i-1} as conversation history before turn i . For token sequence x , s_{x} , and e_{x} represent starting and ending position IDs. The notation x\rightarrow\mathcal{A}(\cdot) indicates sequences that x attends to, and \mathcal{L}(\cdot) denotes language modeling loss (detailed in Appendix A.1 ). 2 Single Pass Fine-tuning on Multi-Turn Reasoning In this section, we highlight the challenges associated with fine-tuning language models on multi-turn reasoning datasets. We present an optimized approach to process an entire conversation in a single forward pass. In multi-turn reasoning data, response tokens r_{i} must attend to reasoning tokens t_{i} during the generation of a_{i} . However, these reasoning tokens must not be visible during subsequent generation of assistant messages a_{j>i} . As a result, it is not possible to construct a single static attention mask that supports both conditions in a conversation within a single forward pass—a capability that is often feasible with non-reasoning datasets. 2.1 N-Pass Approach A straightforward solution is to perform a separate forward pass for every turn (\mathcal{H}_{<i},h_{i},a_{i}) of a given conversation c . While functionally correct, this approach is computationally inefficient: a conversation with N assistant turns results in N separate training examples. Consequently, the effective size of the dataset increases from |\mathcal{D}| to |\mathcal{D}|\times N , inflating training time proportionally. Fig. 1 (a) shows causal attention mask at the time of generation of i th turn response tokens, and Fig. 1 (b) shows causal attention mask for i th turn response tokens when they are part of context during j>i turns. 2.2 1-Pass Approach The primary challenge in applying a single forward pass during training due to discrepancy in the attention behavior of r_{i} can be illustrated as follows 3 3 3 For ease of understanding, we omit the detail that each token within a token sequence also attends to all its preceding tokens, which must be encoded in the attention mask. : \displaystyle r_{i}\rightarrow\begin{cases}\mathcal{A}(\mathcal{H}_{<i},h_{i},t_{i})&\text{generation}\\ \mathcal{A}(\mathcal{H}_{<i},h_{i})&\text{context}\end{cases} We can resolve this issue through the following steps: Duplicating response tokens of each assistant message. We duplicate the response tokens of each assistant message so that one sequence ( r_{i}^{out} ) is used during generation and attends to its associated reasoning tokens. In contrast, the other sequence ( r_{i}^{in} ) is used only as context and does not attend to reasoning tokens. Custom Attention Mask. Duplication of response tokens makes it possible to have a single attention mask that satisfies visibility constraints. We define a custom masking strategy for each type of token sequence (h_{i},t_{i},r_{i}^{\text{in}},r_{i}^{\text{out}}) , ensuring that each token only attends to the appropriate subsequence: \displaystyle h_{i} \displaystyle\rightarrow\mathcal{A}(\mathcal{H}_{<i}^{\textit{in}}) \displaystyle\qquad r_{i}^{in} \displaystyle\rightarrow\mathcal{A}(\mathcal{H}_{<i}^{\textit{in}},h_{i}) \displaystyle t_{i} \displaystyle\rightarrow\mathcal{A}(\mathcal{H}_{<i}^{\textit{in}},h_{i}) \displaystyle\qquad r_{i}^{out} \displaystyle\rightarrow\mathcal{A}(\mathcal{H}_{<i}^{\textit{in}},h_{i},t_{i}) Figure 1 : Causal Attention Masks for N-Pass Approach represents non-zero attention. (a) Attention Mask for generation of response tokens. (b) Attention Mask when response tokens are in context. Assigning Consistent Position IDs. After duplication of response tokens, we need to assign consistent position IDs to tokens to maintain the correct relative positions—as if multiple forward passes were performed for each turn in the conversation. If they are assigned sequentially, or the duplicated assistant response tokens share the same position IDs, it will lead to incorrect relative positions. We need a strategic way of assigning position IDs. The following assignment of the first position ID for each token sequence ensures the relative positions are correct and equivalent to N-Pass approach 4 4 4 Position IDs are assigned sequentially based on the order of tokens within each sequence. : s_{t_{i}}=s_{r^{in}_{i}}=e_{h_{i}}+1\hskip 15.00002pts_{r^{out}_{i}}=e_{t_{i}}+1\hskip 15.00002pts_{h_{i+1}}=e_{r^{in}_{i}}+1 Label Mask. Duplication of the response tokens also raises the question of which tokens should be included in the loss calculation. The following label mask outlines the inclusion criteria for each token type: h_{i}\leftarrow\text{0}\qquad t_{i}\leftarrow\text{1}\qquad r_{i}^{in}\leftarrow\text{0}\qquad r_{i}^{out}\leftarrow\text{1} Fig. 2 shows custom attention mask for i th turn in the 1-Pass Approach. It combines masks for generation and context from the N-Pass Approach into a single mask with position IDs and

Read the complete text