Rhetorical Questions in LLM Representations: A Linear Probing Study

Louie Hong Yao, Vishesh Anand, Yuan Zhuang, Tianyu Jiang · 2026

Read Rhetorical Questions in LLM Representations: A Linear Probing Study by Louie Hong Yao, Vishesh Anand, Yuan Zhuang, Tianyu Jiang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-14128.txt.

Introduction

Yao and three colleagues probe how large language models hold rhetorical questions, asked to persuade rather than to seek facts. With simple linear probes on two social media collections, they find such questions separate cleanly from genuine questions and stay visible across collections, scoring roughly 0.7 to 0.8. Yet probes trained on different collections rank examples differently, so the team argues several distinct patterns, not one shared code, carry rhetorical force.

Catalog record

CREATORLouie Hong Yao, Vishesh Anand, Yuan Zhuang, Tianyu Jiang
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53396
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.14128

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Rhetorical Questions in LLM Representations: A Linear Probing Study Louie Hong Yao, Vishesh Anand, Yuan Zhuang, Tianyu Jiang Abstract Rhetorical questions are asked not to seek information but to persuade or signal stance. How large language models internally represent them remains unclear. We analyze rhetorical questions in LLM representations using linear probes on two social-media datasets with different discourse contexts, and find that rhetorical signals emerge early and are most stably captured by last-token representations. Rhetorical questions are linearly separable from information-seeking questions within datasets, and remain detectable under cross-dataset transfer, reaching AUROC around 0.7-0.8. However, we demonstrate that transferability does not simply imply a shared representation. Probes trained on different datasets produce different rankings when applied to the same target corpus, with overlap among the top-ranked instances often below 0.2. Qualitative analysis shows that these divergences correspond to distinct rhetorical phenomena: some probes capture discourse-level rhetorical stance embedded in extended argumentation, while others emphasize localized, syntax-driven interrogative acts. Together, these findings suggest that rhetorical questions in LLM representations are encoded by multiple linear directions emphasizing different cues, rather than a single shared direction. 1 Introduction Rhetorical language is a common and important part of everyday communication. One of its most typical forms is the rhetorical question, which speakers use not to seek information, but to persuade, challenge, or signal stance. For example, the question “Do you really believe that?” is often used to express doubt rather than to elicit an answer, and “Who would ever agree with this?” functions as a critique rather than a genuine inquiry. In contrast to domains involving formal problem-solving or factual inquiry, which rely on literal interpretation, rhetorical questions convey non-literal meaning shaped by context, speaker intent, and discourse structure. As a result, understanding rhetorical questions provides an important perspective on how large language models interpret and generate language in real-world communicative settings. While rhetorical and informational questions have been studied computationally for many years ( 3 ; 20 ; 33 ; 16 ) , how large language models internally represent rhetorical questions has received far less attention. Existing work on rhetorical question understanding has largely framed the problem as a question-answering or classification task with explicit labels ( 13 ) . While convenient, this formulation focuses on prediction accuracy, whereas our work takes a step forward in understanding the representational basis of rhetorical behavior within the model. To achieve this, we adopt a perspective that examines how these models encode rhetorical question within their internal representational space . Using multiple linear probes ( 21 ) , we investigate where rhetorical signals emerge, whether probes learned within the same dataset capture similar structure, and how probes learned from different data sources compare and transfer across datasets. We find that rhetorical content is linearly separable from information-seeking questions within a given context and remains detectable under cross-dataset transfer. However, probing directions differ substantially within and across datasets. These differences reflect distinct rhetorical properties and lead to different rankings on the same corpus. This shows that strong probing accuracy or transferability does not imply that a property is captured by a single shared representational direction. Instead, rhetorical questions are not organized along a single linear axis, but reflect multiple linguistic features that are emphasized differently depending on context and data. Our code is publicly available. 1 1 1 https://github.com/ruyi101/rq-representation-probing We summarize our contributions as follows: 1. We present a systematic, data-driven analysis of rhetorical question in LLM representations across real-world contexts, showing that rhetorical signals emerge early and are most stably captured by last-token representations in decoder-only models. 2. We reveal a divergence between discriminative performance and representational alignment: although rhetorical questions are consistently linearly separable and transferable, probes learned from different data distributions induce substantially different rankings with little overlap in the upper and lower ends of the ranking. 3. Through qualitative analysis, we demonstrate that rhetorical meaning is inherently heterogeneous, spanning discourse-level rhetorical stance and localized, syntax-driven interrogative acts rather than a single, unified representational dimension. 2 Related Work Rhetorical Questions. Rhetorical questions have long been studied in linguistics and NLP, with early work focusing on their pragmatic and discourse functions ( 15 ; 6 ; 25 ; 10 ; 27 ) . More recent computational studies frame rhetorical questions as a classification or detection task using linguistic features and contextual cues ( 3 ; 20 ; 33 ; 16 ) . Recently, 13 examined rhetorical questions in the context of large language models using QA-style judgments of whether a question in context is rhetorical or informational. Beyond rhetorical questions specifically, recent work has examined rhetorical behavior in LLMs at broader stylistic or strategic levels. 22 introduce a counterfactual framework for measuring rhetorical style independently of substantive content, and 14 develop LLM-based models for annotating rhetorical strategies across domains. In a different vein, 23 analyze grammatical and rhetorical variation in LLM-generated text at the level of model outputs. By contrast, our work focuses on rhetorical questions at the level of internal representations, examining how rhetorical intent is encoded, organized, and transferred across layers and datasets. Related work on sarcasm and irony ( 32 ; 17 ) likewise addresses non-literal language, but centers on distinct pragmatic phenomena and does not directly examine the representation of rhetorical question intent in LLMs. Interpretability of Representations. Linear probing, which evaluates whether a target property is linearly decodable from a model’s internal representations, has long been used to analyze neural networks ( 2 ) . A broad line of work studies neural language models through interpretability and probing methods, using linear probes as diagnostic tools to assess whether linguistic or semantic properties are accessible from model representations. Recent work emphasizes population-level and training-free approaches, such as diffMean directions ( 18 ; 29 ) , as well as sparse autoencoders ( 4 ; 7 ; 5 ; 11 ) that linearly decompose activations into interpretable feature directions. Complementary geometric and information-theoretic analyses have also been applied to study representation spaces in LLMs ( 12 ; 26 ) . Related work has also increasingly explored causal intervention-based methods ( 19 ; 8 ) , such as patching and model editing, to test how specific internal states contribute to downstream behavior. Our work builds on this representation-centric literature by applying linear probing and geometric analysis to rhetorical question intent across contexts. 3 Methods In this section, we describe the datasets, representation choices, and linear probing framework used in our analysis. 3.1 Datasets We conduct our analysis on two real-world rhetorical question datasets drawn from social media, which differ in domain, annotation protocol, and availability of contextual information. RQ. The RQ dataset introduced by 33 consists of Twitter questions annotated as rhetorical or informational. The dataset contains 4,997 instances, with 2,332 labeled as rhetorical, and is split into 3,200 training, 797 validation, and 1,000 test samples. Each instance includes a target question and a prior tweet providing conversational context. In the main analysis, we focus on the question-with-context formulation, which concatenates the prior tweet and the target question. Under this formulation, instances average 38.9 tokens. 2 2 2 All token counts for both RQ and SRAQ use the Llama-3.3-70B-Instruct tokenizer. SRAQ. The SRAQ dataset proposed by 13 draws from Reddit conversation threads, where

Read the complete text