Let’s Predict Sentence by Sentence

Hyeonbin Hwang Byeongguk Jeon, Seungone Kim Jiyeon Kim Hoyeon Chang Sohee Yang, Seungpil Won Dohaeng Lee Youbin Ahn Minjoon Seo · 2025

Read Let’s Predict Sentence by Sentence by Hyeonbin Hwang Byeongguk Jeon, Seungone Kim Jiyeon Kim Hoyeon Chang Sohee Yang, Seungpil Won Dohaeng Lee Youbin Ahn Minjoon Seo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-22202.txt.

Introduction

Hyeonbin Hwang and nine co-authors ask, in 2025 work, whether language models can reason in whole sentences rather than word bits. Models write one small token at a time, while people think in claims and ideas. The team tests if a trained model can be lifted to work in that higher space.

Their frame predicts the code of the next sentence, step by step, instead of the next token. Two code types are tried: one that keeps surface sense, and one trained to look ahead. Guessing can run by turning each code back to text, or by staying in code space the whole way. The look-ahead codes, run in space, matched step-by-step text reasoning across maths, logic, and planning tests, at about half the computing cost. A lens tool shows the hidden steps as plain sentences.

Catalog record

CREATORHyeonbin Hwang Byeongguk Jeon, Seungone Kim Jiyeon Kim Hoyeon Chang Sohee Yang, Seungpil Won Dohaeng Lee Youbin Ahn Minjoon Seo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-22202

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Let’s Predict Sentence by Sentence Hyeonbin Hwang Byeongguk Jeon, Seungone Kim Jiyeon Kim Hoyeon Chang Sohee Yang, Seungpil Won Dohaeng Lee Youbin Ahn Minjoon Seo Abstract Autoregressive Language Models (LMs) generate one token at a time, yet human reasoning operates over higher-level abstractions—sentences, propositions, and concepts. This contrast raises a central question: can LMs likewise learn to reason over structured semantic units rather than raw token sequences? In this work, we investigate whether pretrained LMs can be lifted into such abstract reasoning spaces building on their learned representations. We present a framework that adapts a pretrained token-level LM to operate in sentence space , by autoregressively predicting continuous embeddings of next sentences. We explore two embedding paradigms inspired by classical representation learning: semantic embeddings, learned via autoencoding to preserve surface meaning; and (ii) contextual embeddings, trained via next-sentence prediction to encode anticipatory structure. We evaluate both under two inference regimes: Discretized , which decodes each predicted embedding into text before re-encoding; and Continuous , which reasons entirely in embedding space for improved efficiency. Across four domains—mathematics, logic, commonsense, and planning—contextual embeddings under continuous inference show competitive performance with Chain-of-Thought (CoT) while reducing inference-time FLOPs in average by half. We also present early signs of scalability and modular adaptation. Finally, to visualize latent trajectories, we introduce SentenceLens , a diagnostic tool that decodes intermediate model states into interpretable sentences. Together, our results indicate that pretrained LMs can effectively transition to abstract, structured reasoning within latent embedding spaces. ∗ ∗ ∗ Our code is available here . ${}^{*}$ ${}^{*}$ footnotetext: Equal contribution 1 Introduction Autoregressive Language Models (LMs) have achieved remarkable success on complex reasoning tasks through a simple objective: Next-Token Prediction [ 1 ] . This success is further amplified by Chain-of-Thought (CoT), which generates explicit intermediate reasoning steps to guide the model [ 2 ] . Recent advancements demonstrate substantial gains in performance by scaling inference-time computation even further [ 3 , 4 ] . However, next-token prediction requires generating long reasoning chains one token at a time, making it computationally inefficient. Also, it remains unanswered whether reasoning at such granularity is genuinely optimal. Figure 1 : Sentence-level reasoning framework. Training : the latent model reads the question tokens and previous embeddings, predicts \hat{h}_{t} , and a frozen decoder reconstructs s_{t} ; Inference : embedding can be rolled forward by (a) Discretized : decode \rightarrow text \rightarrow encode or (b) Continuous : pass-through. While token-level generation has driven recent progress, human cognition typically operates over higher-level abstractions—such as concepts, propositions, or full sentences [ 5 , 6 , 7 ] . Prior works suggest that language models may similarly benefit from operating at these higher levels, potentially enabling more structured and computationally efficient reasoning [ 8 , 9 ] . In this paper, we investigate whether pretrained language models can effectively build higher-level representations directly by abstracting over their existing token-level representations, without the prohibitive cost of pre-training from scratch. Specifically, we introduce a framework that repurposes pretrained next-token Transformers to reason in a latent sentence-level embedding space. Instead of producing outputs token-by-token, our approach predicts continuous embeddings for entire sentences, which can be decoded back into natural language yet primarily function as abstract conceptual representations. To systematically explore viable latent representations, we draw inspiration from the well-established dichotomy in classical representation learning between reconstruction-based and prediction-based methods [ 10 , 11 , 12 ] . We define two embedding paradigms: (1) Semantic embeddings, which prioritize preserving textual fidelity through autoencoding, and (2) Contextual embeddings, which focus on capturing predictive context via next-sentence prediction. We evaluate models trained with these embeddings under two inference regimes: Discretized , which decodes each predicted embedding into natural language before re-encoding it as the next input, and Continuous , which performs reasoning entirely within the continuous embedding space. Our empirical findings demonstrate that contextual embeddings consistently outperform semantic embeddings across diverse reasoning domains including mathematics, logic, commonsense, and planning tasks. Notably, contextual embeddings using Continuous inference show competitive performance to token level Chain of Thought reasoning while reducing inference time computational cost by half in average. Finally, we introduce SentenceLens , a diagnostic tool that translates intermediate hidden states into readable sentences, thus providing intuitive transparency into the model’s internal reasoning trajectories. Overall, our analysis provides initial evidence that pretrained inductive biases acquired from token level modeling can be effectively adapted to structured, abstraction level reasoning within latent embedding spaces. Figure 2 : Illustration of the different types of sentence embeddings used in our framework. 2 Sentence embeddings for autoregressive modeling Unsupervised and semi-supervised sequence representation learning has predominantly evolved along two primary paradigms: reconstruction-based and prediction-based methods [ 10 , 11 , 12 ] . Both methodologies have demonstrated strong empirical performance, yet each emphasizes distinct representational strengths. Reconstruction-based methods, typically employing autoencoder architectures, excel at semantic fidelity by explicitly encoding and reconstructing input sequences [ 10 ] , whereas prediction-based methods prioritize capturing contextual semantics by modeling relations to subsequent sequences [ 11 ] . Previous research suggests that the optimal embedding strategy varies significantly depending on the target application [ 13 ] . In this light, we systematically explore both embedding paradigms within the context of sentence-level autoregressive modeling. Specifically, we adapt an autoregressive Language Model autoencoder framework to construct and evaluate two distinct embedding approaches: semantic embedding , derived through reconstruction objective, and contextual embedding , derived through predictive objective. 2.1 Sentence embedding construction To ensure scalability and avoid vocabulary constraints inherent to discrete codebooks [ 14 ] , we utilize a continuous embedding space. This approach facilitates flexible representational capacity scaling with embedding dimensionality [ 15 ] . We build upon the autoencoding framework proposed by ICAE [ 16 ] and adapt a decoder-only Transformer (e.g., GPT-2), employing shared parameters for encoding and decoding: \theta_{\text{ENC}}=\theta_{\text{DEC}} . Given an input sequence x=(x_{1},\dots,x_{N}) , the encoder produces a sequence of hidden states H=(h_{1},\dots,h_{N}) . We then define the embedding h^{[-1]}:=h_{N} as the latent representation of the entire input sequence. This embedding conditions the decoder, trained autoregressively with cross-entropy loss: \hat{y}=\theta_{\text{DEC}}(h^{[-1]})\quad\text{and}\quad\mathcal{L}_{\text{CE}}=-\sum_{t=1}^{N}\log p(y_{t}\mid y_{<t},h^{[-1]}) Note that most reasoning tasks consist of a question or instruction q , followed by an ordered sequence of reasoning steps (s_{1},\dots,s_{n}) . In this light, we construct training examples tailored to each embedding type as follows (See Figure 2 ): Semantic embeddings. Each reasoning step s_{i} independently forms the input and reconstruction target x=y=s_{i} . Training this way ensures the embedding h^{[-1]} encapsulates complete and detailed semantics of the individual reasoning step. Contextual embeddings. We form context–target pairs, where context x includes the question and preceding reasoning steps (q,s_{1},\dots,s_{i-1}) , and the target is the current step y=s_{i} . Thus, embeddings must capture predictive cues essential for reasoning step generation. Reconstruction Prediction Dataset Semantic (EM) CTX-B CTX-C CoT GSM8K 98.5 42.0 42.1 43.4 CSQA 98.5 33.8 35.1 35.7 ProsQA 100.0 80.2 75.3 77.5 Blocksworld 100.0 89.9 90.1 84.3 Table 1: Performance of Semantic and Contextual Embeddings across datasets. For Semantic embeddings, we report exact match (EM). For Contextual embeddings, we compare final-answer accuracy (ACC) under different decoding schemes: CTX-B (unregularized), CTX-C (contrastive), and CoT (language-level chain-of-thought). Optionally, to bridge semantic fidelity with predictive abstraction, we also try a

Read the complete text