Exploring the Hidden Capacity of LLMs for One-Step Text Generation Gleb Mezentsev, Ivan Oseledets Abstract A recent study showed that large language models (LLMs) can reconstruct surprisingly long texts – up to thousands of tokens – via autoregressive generation from just one trained input embedding. In this work, we explore whether autoregressive decoding is essential for such reconstruction. We show that frozen LLMs can generate hundreds of accurate tokens in just one token-parallel forward pass, when provided with only two learned embeddings. This reveals a surprising and underexplored multi-token generation capability of autoregressive LLMs. We examine these embeddings and characterize the information they encode. We also empirically show that, although these representations are not unique for a given text, they form connected and local regions in embedding space – suggesting the potential to train a practical encoder. The existence of such representations hints that multi-token generation may be natively accessible in off-the-shelf LLMs via a learned input encoder, eliminating heavy retraining and helping to overcome the fundamental bottleneck of autoregressive decoding while reusing already-trained models. 1 Introduction Large language models are typically trained to generate text in an autoregressive manner – they predict one token at a time based on the previously generated context. Several attempts aim to change this. However, they either require an additional model for candidate generation ( Leviathan et al., 2023 ) , substantial additional fine-tuning of autoregressive LLM ( Cai et al., 2024 ; Stern et al., 2018 ; Gloeckle et al., 2024 ) or full model retraining ( Ghazvininejad et al., 2019 ; Austin et al., 2021 ; Li et al., 2022 ) . This leaves an open question – is it possible to reuse autoregressively pretrained LLM for multi-token generation with minimal to no additional training. We discover a previously undocumented phenomenon, that can help us to achieve this goal. We found that for any given text of reasonable length, there exists a latent one-vector representation of this text, such that, if a frozen pretrained LLM is conditioned on this representation, it accurately generates the whole text in a single forward pass, without any iterative decoding. In this work, we demonstrate this phenomenon, investigate what those compressed representations encode and whether this finding reveals anything about LLMs’ parallel generation capabilities. Figure 1: One pass, many tokens. Each dot shows the maximum exact reconstruction length in a single non-autoregressive forward pass with frozen weights, conditioned only on two learned embeddings – evidence of hidden multi-token capabilities. Our contribution is as follows: 1. We show that LLMs can reconstruct arbitrary sequences of hundreds of tokens from as few as two learned input embeddings, with one of them being universal for all texts. 2. We identify key design aspects for such a setup, that enable this generation, including the critical importance of input token arrangement. 3. We study how the reconstruction capability varies with the model size and the nature of the target sequence (e.g. natural vs synthetic text). 4. We empirically characterize learned representations – analyze their information content and embedding-space geometry. The code is available at this GitHub page . 2 Related Work The most direct influence for our work is a paper by Kuratov et al. (2025) , which showed that frozen LLM can reconstruct an arbitrary sequence of tokens T=[t_{1},\dots,t_{N}] if given a sequence of special, so-called memory tokens [mem_{1},\dots,mem_{K}] . The embeddings for these tokens are trained by optimizing a causal language modeling objective over a concatenated input sequence {Z=[mem_{1},\dots,mem_{K},t_{1},\dots,t_{N}]} passed through a frozen LLM. In the case of perfect next-token prediction accuracy (which could be achieved for reasonable text length), this allows the model to autoregressively predict the whole text starting from the memory tokens. The number of memory tokens controls the maximum text length and can be as low as one. Although surprisingly long (up to 1568 tokens) texts could be compressed even into a single memory token, the authors note that the embeddings trained from different random initializations for the same text end up far apart. Moreover, linear interpolations between those embeddings produce poor reconstruction accuracy, suggesting that the solution space lacks desirable smoothness and locality qualities, which are important for learning a practical encoder that could replace direct optimization. Our work also relates to efforts in prompt-tuning and its variants ( Lester et al., 2021 ; Liu et al., 2024 ; Li and Liang, 2021 ) . Most similarly, Lester et al. (2021) train task-specific soft tokens to condition frozen LLMs to improve their performance on new tasks. Several speculative ( Xia et al., 2023 ) and parallel ( Santilli et al., 2023 ) decoding approaches utilize a similar mechanism for multiple token prediction using decoder architectures. More specifically, they add special [PAD] or [MASK] tokens at the end of the current context in order to make a prediction for several tokens into the future at once. Critically, in these works either special training or multiple generative iterations are required. Unlike prior work, we show that a frozen LLM can generate accurate multi-token sequences in one forward pass without additional LLM training or iterative decoding. 3 Method To adopt the approach from Kuratov et al. (2025) to the non-autoregressive case, we replace all input tokens of the LLM with specially trained "proto-tokens" and predict the target token sequence in one forward pass. In practice, "proto-tokens" are just trainable vectors that are not tied to any real items in the vocabulary. The main difference between regular tokens and these "proto-tokens" is that "proto-tokens" encode multiple tokens at once and only produce human-readable text after passing through the LLM. Our goal is to identify the smallest possible number of such "proto-tokens" needed for accurate reconstruction. Interestingly, we find that it is essential to have at least two – the performance drops dramatically when using only one (see Section 4 ). There are many ways to arrange two vectors into an input sequence of arbitrary length. We report results for different variants later in the paper, but here we describe the arrangement that is used in the majority of the experiments. Exact scheme Figure 2: Two "proto-tokens" (trainable embeddings) are fed into frozen, pretrained LLM and optimized in such a way that LLM predicts an arbitrary token-sequence in a single forward pass. e_{t} is trained for each text separately, while m could be shared across texts. We introduce two "proto-tokens" e and m with trainable embeddings of dimension d_{model} (model input embedding dimension) and construct the input sequence as follows: Z=[e,m,m,\dots,m] – one copy of token e is followed by N-1 copies of token m , where N is the target text length. We then train the vectors by optimizing cross-entropy loss between the target sequence T=[t_{1},t_{2},\dots,t_{N}] and the frozen LLM’s output for the input sequence: L_{CE}=-\sum_{i=1}^{N}log\mathbb{P}_{LM}(t_{i}\mid e,\underbracket{m,\dots,m}_{i-1}) (1) The prediction is obtained using standard causal attention masking, so that the predicted probabilities for the token t_{i} depend on the first i input "proto-tokens" (see Figure 2 ). Metrics Our main evaluation metric is the number of correctly reconstructed tokens in a generated sequence, defined as: C_{tokens}=\sum_{i=1}^{N}\mathbbm{1}(LM(Z_{[1:i]})=t_{i}) (2) Additionally, we measure the amount of information contained in the reconstructed token sequence from the perspective of causal language modeling with a given LLM. Specifically, we compute the cross-entropy between the compressed sequence and LLM’s autoregressive probability distribution: H_{LM}=-\sum_{i=1}^{N}log\mathbb{P}_{LM}(t_{i}\mid t_{