Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling

Yuan Yidi · 2026

Read Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling by Yuan Yidi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-18464.txt.

Introduction

Yuan applies semantic tube prediction at thinking-step boundaries to sharpen multi-step latent guessing in language models. The first recipe samples random token spans, pushing hidden-state trajectories toward locally linear paths. Sampling instead at back-to-back meaning-step edges yields 168 times sharper multi-step latent guesses than frozen rivals on ProcessBench. A three-layer net cuts error further over straight-line guesses, showing paths form smooth curves rather than straight lines, with sampling spot as the key variable.

Catalog record

CREATORYuan Yidi
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30289
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.18464

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Step Prediction: Multi-Step Latent Forecasting in LLM Reasoning Trajectories via Step Sampling Yuan Yidi Abstract Semantic Tube Prediction (STP) leverages representation geometric to regularize LLM hidden-state trajectories toward locally linear geodesics during fine-tuning, thereby greatly improving data efficiency. The original STP recipe samples random token sub-spans, which is compatible with the base large language model (LLM) training architecture. Inspired by STP, we are interested to investigate whether the sampling position can further enhance the semantic structure of multi-step reasoning, and hence affect its geometric impact. We applied STP at consecutive semantic reasoning step boundaries and achieved 168 × 168\times more accurate multi-step latent prediction than frozen baselines on ProcessBench (3,400 samples), compared to only 4 × 4\times for the random-token STP. Probing the latent manifold with a learned non-linear predictor reveals that STP-shaped trajectories are smooth curves, not straight lines: a 3-layer MLP reduces prediction error by a further 3 3 – 12 × 12\times over linear extrapolation on step-boundary models. Removing the language modeling loss yields trajectories that are 2 × 2\times more MLP-predictable than the combined loss, revealing a tradeoff between generation quality and geometric purity. Our results identify sampling position as the critical variable in geometric regularization and establish multi-step latent prediction MSE as a new evaluation metric for this class of methods. 1 Introduction Large Language models (LLMs) reason by generating step-by-step solutions, producing many tokens per reasoning step. Each token requires a full autoregressive forward pass, making multi-step reasoning computationally expensive. A growing body of work explores latent reasoning where computation is performed in the model’s continuous hidden-state space rather than decoding discrete tokens at every step ( Hao et al. 2024 ; PLaT 2026 ) . For latent reasoning to be feasible, the model’s hidden-state trajectory must be predictable, which is given the current reasoning state, future states must be forecastable with reasonable accuracy. Semantic Tube Prediction (STP) ( Huang et al. 2026 ) provides a theoretical and practical framework for this. Building on the Geodesic Hypothesis, which insists that error-free token sequences trace locally linear geodesics on a smooth semantic manifold, STP adds a supplementary cosine-displacement loss during fine-tuning that forces consecutive hidden-state displacement vectors to be parallel. The resulting “Semantic Tube” confines trajectories to a tubular neighborhood of the geodesic, improving signal-to-noise ratio and enabling 16\times data efficiency gains. STP samples its loss at random token sub-spans, where three indices s<r<t are drawn uniformly from the token sequence. While this method is elegant, we would like to explore if the semantic structure of the reasoning process, which is the natural boundaries where the model transitions from one reasoning step to the next, can contribute to the geometric smoothness of the method. We hypothesize that aligning the geometric regularization with the semantic structure of reasoning at step boundaries rather than at arbitrary token positions will produce more predictable trajectories. We test whether step-boundary STP creates the latent-space conditions that future JEPA ( LeCun 2022 ) or COCONUT-style systems would need, as measured via post-hoc multi-step prediction accuracy of both linear and learned predictors. Our key contributions. 1. Sampling position is the critical variable. Applying STP at consecutive step boundaries achieves 168\times prediction improvement vs. 4\times for random-token STP—a 40\times gap from where the loss is sampled, not what the loss computes (§ 5.1 ). 2. Multi-step latent prediction MSE as evaluation metric. We introduce and validate this metric with decoding fidelity tests: predicted embeddings decode to the correct next token 93.4% of the time (§ 5.4 ). 3. Trajectories are smooth curves, not straight lines. A learned MLP predictor reduces error by 3 – 12\times over linear on STP-shaped models but finds no systematic structure on baselines, revealing three distinct manifold regimes (§ 6.1 ). 4. \mathcal{L}_{\text{NTP}} creates a tradeoff. Removing the language modeling loss yields 2\times better MLP prediction at the cost of 3.7 pp GSM8K accuracy—generation quality vs. geometric purity (§ 5.1 , § 6.1 ). 5. Negative finding. Geometric smoothness does not encode step correctness (AUC \approx 0.5 ; § A.2 ): the Semantic Tube captures organized thinking, not correct thinking. 2 Background 2.1 Semantic Tube Prediction Huang et al. 2026 propose the Geodesic Hypothesis, which describes that token sequences generated by LLMs trace geodesics on a smooth semantic manifold, and deviations from these geodesics represent noise. They formalize this as the STP loss: \mathcal{L}_{\text{STP}}=\mathbb{E}_{s<r<t}\left[1-\cos(h_{r}-h_{s},\;h_{t}-h_{r})\right] (1) where h_{s},h_{r},h_{t} are hidden states at randomly sampled token positions. Minimizing \mathcal{L}_{\text{STP}} forces consecutive displacement vectors to be parallel, confining trajectories to a tube around the geodesic. Combined with the standard next-token prediction loss, the training objective is \mathcal{L}=\mathcal{L}_{\text{NTP}}+\lambda\cdot\mathcal{L}_{\text{STP}} . With the predictability of a geometric constraint, STP is able to dramatically reduce the amount of data required for training. Finetuning with 16\times less training data across multiple model families ( Huang et al. 2026 ) resulted in comparable results with full-data training. 2.2 Joint Embedding Predictive Architectures We draw motivation from the JEPA paradigm ( LeCun 2022 ) , which advocates predicting in latent space rather than reconstructing inputs. I-JEPA ( Assran et al. 2023 ) and V-JEPA ( Bardes et al. 2024 ) demonstrate this for images and video; LLM-JEPA ( Huang et al. 2025 ) extends it to language. Interestinly, STP was interestingly introduced to solve the two-view problem of LLM-JEPA, which requires the text data to be paired with the code form of the same information. We do not implement a JEPA architecture as our training lacks target encoder, learned predictor module, and continuous embedding feedback loop. Instead, we test the preconditions that the latent space supports accurate multi-step prediction. 2.3 Geometric Analysis of LLM Hidden States Several works analyze the geometry of LLM hidden states diagnostically. Wang et al. 2025 measure curvature across layers; Zhou et al. 2026 apply Menger curvature to reasoning trajectories; Herrmann et al. 2025 introduce PHi loss and show that hidden-state unpredictability correlates with correct solutions on difficult problems. Jiang et al. 2026 decompose reasoning traces into Progress (displacement magnitude) and Stability (curvature) at the token level. Sun et al. 2026 extract activations at step markers and find that correct and incorrect solutions diverge geometrically at late reasoning steps. Carson & Reisizadeh 2025 model reasoning trajectories as drift-diffusion systems with regime switching. Damirchi et al. 2026 find that smooth layer-wise trajectories correlate with valid reasoning. 2.4 Latent Reasoning Latent reasoning to reduce or avoid token-wise decoding has been explored in several recent works. COCONUT ( Hao et al. 2024 ) feeds the last hidden state back as the next input embedding, enabling latent iteration without token decoding. PLaT ( PLaT 2026 ) decouples reasoning from verbalization entirely. CoLaR ( Tao et al. 2025 ) trains a latent head with a next-compressed-embedding objective at random token positions. LightThinker ( Zhang et al. 2025 ) compresses intermediate steps into gist tokens at step boundaries. Teoh et al. 2025 train a next-latent prediction head with Smooth L1 loss at every token, supporting multi-step rollouts for world modeling. At inference time, The Geometric Reasoner ( Zhuang et al. 2026 ) scores candidates via bumpiness penalties at chunk boundaries, and STEP ( Sun et al. 2026b ) trains a classifier on step-boundary hidden states to detect reasoning errors. 3

Read the complete text