Better Language Model Inversion by Compactly Representing Next-Token Distributions
Murtaza Nazir, Matthew Finlayson, John X. Morris, Xiang Ren, Swabha Swayamdipta · 2025
Read Better Language Model Inversion by Compactly Representing Next-Token Distributions by Murtaza Nazir, Matthew Finlayson, John X. Morris, Xiang Ren, Swabha Swayamdipta (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-17090.txt.
Introduction
Murtaza Nazir, Matthew Finlayson, John Morris, and a wider team wrote this machine learning paper, published in 2025. It recovers the hidden prompts fed to a language model, including secret system messages, from nothing but the model's own output probabilities.
Prompt inversion matters because a leaked system message can expose private instructions. The key insight is geometric: a model's next-token probability vectors live in a low-dimensional subspace. So the full distribution across many generation steps can be compressed without loss by a linear map, preserving far more evidence for the detective work. The resulting method multiplied exact recovery rates two to three and a half times over prior art, in one case from seventeen to sixty percent. An inverter trained on short generations even improved when given longer ones at test time.
Catalog record
| CREATOR | Murtaza Nazir, Matthew Finlayson, John X. Morris, Xiang Ren, Swabha Swayamdipta |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 36 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-17090
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Better Language Model Inversion by Compactly Representing Next-Token Distributions Murtaza Nazir, Matthew Finlayson, John X. Morris, Xiang Ren, Swabha Swayamdipta Abstract Language model inversion seeks to recover hidden prompts using only language model outputs. This capability has implications for security and accountability in language model deployments, such as leaking private information from an API-protected language model’s system message. We propose a new method— prompt inversion from logprob sequences (PILS)—that recovers hidden prompts by gleaning clues from the model’s next-token probabilities over the course of multiple generation steps. Our method is enabled by a key insight: The vector-valued outputs of a language model occupy a low-dimensional subspace. This enables us to losslessly compress the full next-token probability distribution over multiple generation steps using a linear map, allowing more output information to be used for inversion. Our approach yields massive gains over previous state-of-the-art methods for recovering hidden prompts, achieving 2–3.5 times higher exact recovery rates across test sets, in one case increasing the recovery rate from 17% to 60%. Our method also exhibits surprisingly good generalization behavior; for instance, an inverter trained on 16 generations steps gets 5–27 points higher prompt recovery when we increase the number of steps to 32 at test time. Furthermore, we demonstrate strong performance of our method on the more challenging task of recovering hidden system messages . We also analyze the role of verbatim repetition in prompt recovery and propose a new method for cross-family model transfer for logit-based inverters. Our findings show that next-token probabilities are a considerably more vulnerable attack surface for inversion attacks than previously known. 1 Introduction The task of language model inversion is to recover an unknown prefix string (hidden prompt), given only information about a language model’s 1 1 1 In this work, we only concern ourselves with causal language models as inversion targets. outputs, conditioned on that prefix. This capability can potentially be used to steal hidden prompts, leak private information, or (on the flip side) detect malicious prompts that could cause harmful behavior in language models. Advancements in inversion, thus have important implications for language model security and accountability. Prior work in language model inversion leverages information in next-token (log-) probabilities —colloquially known as logprobs — [ 21 ] , text outputs [ 34 , 12 ] , or employing prompt-based attacks [ 35 ] . However, these methods have shown only modest success. For example, state-of-the-art methods recover fewer than one-in-four Llama 2 Chat prompts from in-distribution evaluation sets, and fare much worse on out-of-distribution prompts. This work aims to improve the performance and generalizability of language model inversion, with a focus on logprobs-based inversion, since logprobs contain rich information about model outputs. Surprisingly, the best-known logprobs-based method, Logit2Text or L2T [ 21 ] , lags behind more recent text-based inversion methods [ 34 ] . Notably, L2T only uses language model outputs from a single generation step, since logprobs are expensive to obtain from typical language model APIs and require a lot of space—each logprob is a vector of dimension equal to the vocabulary size of the target model, which can be hundreds of thousands of tokens. We propose a method to overcome the high representation size and API costs of L2T. As illustrated in Figure 1 , we apply a lossless compression to the target model’s logprob outputs (at multiple generation steps) to obtain compact representations with dimension equal to the target model’s embedding size D . We confirm empirically that these representations are good approximation of the full logprobs, by showing that an inverter that uses them performs as well as L2T (and slightly better). The key insight of our method is that logprobs live in a D -dimensional subspace, meaning that we can compress them with a simple linear map. Furthermore, obtaining these compact representations requires only D logprob values from the target model, greatly reducing the API cost by 1–2 orders of magnitude. With this improved representation scheme, we propose a new inversion method, prompt inversion from logprob sequences (PILS), that incorporates target model outputs from multiple generation steps as input to our inverter. The intuition behind our approach being effective is that the target model may not surface information about certain parts of the prompt until later in the generation. We find that our method massively improves performance on inversion, and boasts an exact recovery rate 2 – 3.5\times higher than the previous state-of-the-art for both in-domain and out-of-domain prompts. We also find that our trained inverters exhibit surprisingly good generalization: an inverter trained on 16 generation steps continues to improve as we increase the number of steps beyond 16 at test time. Finally, we leverage our compact representations to propose a method to adapt our inverter to new models without any additional training (model transfer), a novel transfer method for logprob-based inverters. Black box Hidden prompt T Logprobs Language model T Inverter input Compress Recovered prompt Inverter Architecture Black box Hidden state \boldsymbol{h} \mathbb{R}^{D} Logits \mathbb{R}^{V} Unembed \boldsymbol{W} Probabilities \boldsymbol{p} \Delta^{V} softmax alr logits \mathbb{R}^{V-1} alr Inverter input \mathbb{R}^{D} subset Linear Compression Figure 1: Our goal is to recover a hidden prompt based on the outputs of a black box language model. To do this, we take a sequence of T logprobs, losslessly compress them into a sequence of T low-dimensional vectors, and feed them into an encoder-decoder inverter model, which outputs the recovered prompt. Our compression method takes advantage of the fact that model outputs are linear projections of the language model’s D -dimensional final hidden state (see Section 2.1 ). 2 Preliminaries We establish some notation, assumptions, and mathematical background for our method. We assume a typical language model architecture with embedding size D , and vocabulary size V . At every generation step, the model produces a hidden state \boldsymbol{h}\in\mathbb{R}^{D} , which is multiplied by the model’s unembedding matrix \boldsymbol{W} to obtain logits \boldsymbol{\ell}=\boldsymbol{W}\boldsymbol{h}\in\mathbb{R}^{V} , which are normalized via the softmax function to obtain probabilities \boldsymbol{p}=\softmax(\boldsymbol{\ell}) . The entries of \boldsymbol{p} are interpreted as the model’s predicted probability for each token in its vocabulary. Generation typically proceeds by sampling according to the probabilities in \boldsymbol{p} , or by greedily picking the most-probable token at each generation step. 2.1 Language model outputs are losslessly compressible We now show how it is possible to recover the hidden state of a language model from its probability output \boldsymbol{p} up to a linear transformation. This demonstrates exactly how we compress the logprobs of the language model in our proposed method ( Section 3 ). Theorem 1 . If a language model with hidden size D , vocabulary size V , and unembedding matrix \boldsymbol{W} , generates a hidden state \boldsymbol{h} and outputs \boldsymbol{p}=\softmax(\boldsymbol{W}\boldsymbol{h}) , then for any set of indices \mathcal{D}\subseteq\{1,2,\ldots,V\} we have that \alr(\boldsymbol{p})_{\mathcal{D}}\in\mathbb{R}^{D} is a linear transformation of \boldsymbol{h} . Proof. Probability vectors \boldsymbol{p} have the property that all entries are in the range (0,1) and that the entries sum to 1. It is a lesser known fact that the set of valid probability distributions over V items—known as the simplex, or \Delta^{V} —forms a vector space, albeit with non-standard definitions of addition +_{\Delta} and scalar multiplication \cdot_{\Delta} [ 15 ] . In particular, for vectors \boldsymbol{p} and