Unnatural Languages Are Not Bugs but Features for LLMs

Keyu Duan, Yiran Zhao, Zhili Feng, Jinjie Ni, Tianyu Pang, Qian Liu, Tianle Cai, Longxu Dou, Kenji Kawaguchi, Anirudh Goyal, J. Zico Kolter, Michael Qizhe Shieh · 2025

Read Unnatural Languages Are Not Bugs but Features for LLMs by Keyu Duan, Yiran Zhao, Zhili Feng, Jinjie Ni, Tianyu Pang, Qian Liu, Tianle Cai, Longxu Dou, Kenji Kawaguchi, Anirudh Goyal, J. Zico Kolter, Michael Qizhe Shieh (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-01926.txt.

Introduction

Gibberish text that models read fine is a feature, not a flaw, argue Keyu Duan and co-authors in a 2025 arXiv paper. Such strings look like noise to people but hold sense for large language models. The team tests if those hidden traits travel across models and tasks. Fine-tunes on gibberish forms of teach sets also get a check.

Gibberish traits did travel at read time. Models taught on gibberish matched those taught on clean text in fair tests. Reads of model steps show noise stripped first, then sense pulled from what stays. The work reframes odd text as a real tongue for models.

Catalog record

CREATORKeyu Duan, Yiran Zhao, Zhili Feng, Jinjie Ni, Tianyu Pang, Qian Liu, Tianle Cai, Longxu Dou, Kenji Kawaguchi, Anirudh Goyal, J. Zico Kolter, Michael Qizhe Shieh
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE51 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-01926

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Unnatural Languages Are Not Bugs but Features for LLMs Keyu Duan, Yiran Zhao, Zhili Feng, Jinjie Ni, Tianyu Pang, Qian Liu, Tianle Cai, Longxu Dou, Kenji Kawaguchi, Anirudh Goyal, J. Zico Kolter, Michael Qizhe Shieh Abstract Large Language Models (LLMs) have been observed to process non-human-readable text sequences, such as jailbreak prompts, often viewed as a bug for aligned LLMs. In this work, we present a systematic investigation challenging this perception, demonstrating that unnatural languages - strings that appear incomprehensible to humans but maintain semantic meanings for LLMs - contain latent features usable by models. Notably, unnatural languages possess latent features that can be generalized across different models and tasks during inference. Furthermore, models fine-tuned on unnatural versions of instruction datasets perform on-par with those trained on natural language, achieving 49.71 49.71 win rates in Length-controlled AlpacaEval 2.0 in average across various base models. In addition, through comprehensive analysis, we demonstrate that LLMs process unnatural languages by filtering noise and inferring contextual meaning from filtered words. Our code is publicly available at https://github.com/John-AI-Lab/Unnatural_Language . † † affiliationnotice: Equal contribution 1 Introduction Large Language Models (LLMs) ( OpenAI, 2023 ; Touvron et al., 2023 ; Dubey et al., 2024 ; anthropic, 2024 ) have shown remarkable capabilities in understanding and generating human-readable text, achieving impressive performance across tasks, spanning from question answering ( Bisk et al., 2020 ; Ni et al., 2024 ) and mathematical reasoning ( Cobbe et al., 2021 ; Hendrycks et al., 2021 ; Gao et al., 2024 ) to open-ended dialogue ( Li et al., 2023 ) . Such abilities are largely attributed to targeted alignment training ( Wei et al., 2021 ; Ouyang et al., 2022 ) , which post-train models to better follow instructions and adhere to preferred behaviors. Despite being specifically tuned, non human-readable data sometimes can unexpectedly influence model behavior. In computer vision, Ilyas et al. (2019) ; Nguyen et al. (2015) find that seemingly unrecognizable images could be leveraged to train reasonable good image classification models. This phenomenon extends to natural language processing, where Zou et al. (2023) demonstrate that LLMs could also be prompted with an unreadable suffix to generate objectionable outputs, even though LLMs are well-trained for not doing so. Besides, Pfau et al. (2024) discovers that by appending non human-readable filler tokens to the input, LLMs could solve algorithmic tasks more accurately. However, regarding LLMs’ surprising behaviors in response to non human-readable inputs, there lacks systematic studies exploring the properties and applications of such non human-readable strings and interpreting the underlying mechanisms in LLMs. This raises a fundamental question: whether these non human-readable strings are truly devoid of meaning or contain latent features usable by models? Table 1 : Concrete examples of natural contexts’ unnatural version in SycContextQA and SimGSM8K. Natural Context Unnatural Version The stock price of GoldMine Inc. increased by 20% last week. (alt+eqn={\\>; {};The\\,\\stock baaelkrie@nuier priceungeureau got sich last ’#GM;;heidisation Inc. weekestig %}20% durch’),png encrypt render \"OK GoldMine.",preventDefault Carly collected 7 starfish with 5 arms each and one seastar with 14 arms. |Each and : algebra dinner! absolutely 7 do): shortly . seastar collectedthe ‘’ kW)$, one !5 ! 14‘ starfish with sic}}_{\label Carly} arms. Onehorailey constructed WriteStatus($$\Toggle Zwezeichnung OK Figure 1 : (a) : Unnatural languages searching method. (b) : We construct question-answering tasks using unnatural contexts and discover that unnatural languages can be directly transferred across a broader range of tasks and understood by diverse LLMs. (c): LLMs fine-tuned on unnatural versions of instruction datasets perform on-par with those trained on natural language. To answer the above research questions, we study a phenomenon named unnatural languages - strings that deviate from natural language syntax and appear extremely noisy to human readers, yet remain understandable to LLMs. Specifically, as illustrated in Figure 1 (a), we propose an approach to search for a semantically equivalent but syntactically unnatural version of the natural string, where semantic equivalence is established through their ability to be translated back to natural form via models performing translation inference. For the searching process, we employ a gradient-based stochastic sampling procedure to obtain a set of candidate unnatural strings; evaluate their probability of being translated back to the corresponding natural versions across multiple models; and select the unnatural candidate that yields the highest probability. We repeat the process until either convergence occurs or a maximum number of iterations is reached, with the final converged string representing the semantically equivalent unnatural string. We first explore whether these unnatural languages possess latent features that can be generalized across different models and tasks during inference. To investigate this, as shown in Figure 1 (b), we construct several context-based question-answering datasets, where the context is unnatural obtained by the searching approach while natural questions related to the context are provided. Specifically, to prevent models from relying on common-sense memory when answering questions without context, we develop SynContextQA, a synthetic dataset generated by another LLM, containing contexts about non-existent entities paired with corresponding questions. We then transformed natural contexts into unnatural versions using the unnatural language searching method while preserving the original questions. Additionally, to ensure models do not simply extract keywords from unnatural contexts in SynContextQA, we create SimGSM8K, a dataset of simple questions derived from GSM8K ( Cobbe et al., 2021 ) . We chose simple questions to minimize the impact of reasoning ability on our results and focus primarily on unnatural language comprehension. As with SynContextQA, we transformed these contexts into unnatural versions. Table 1 shows concrete examples of two datasets. With these datasets, we test a large variety of LLMs, including open-source models as well as commercial models. The results show that compared to natural context, all models can recover 82.0\% of the original accuracy on our constructed SynContextQA dataset and 61.6\% on SimGSM8K, demonstrating that unnatural languages contain latent features that enable comprehension across different scenarios. Moreover, we explore whether these unnatural languages possess transferable latent features that can be effectively utilized in instruction tuning to improve models’ instruction-following capabilities. Specifically, as shown in Figure 1 (c), we employ a high-quality but small size instruction tuning dataset LIMA ( Zhou et al., 2023 ) , and we replace the original instructions with our equivalent unnatural versions searched using our proposed approach. We show that the models fine-tuned on it and the original have on-par performance on prestigious benchmarks, including Length-controlled (LC) AlpacaEval 2.0 ( Li et al., 2023 ) and MixEval ( Ni et al., 2024 ) . Particularly, on LC AlpacaEval 2.0, the three models — Llama-3-8B ( Dubey et al., 2024 ) , Gemma-2-9B ( Team et al., 2024 ) , and Llama-3-70B ( Dubey et al., 2024 ) — tuned on the unnatural LIMA achieves an winrate of 49.78\% , 47.13\% , and 52.22\% against the corresponding models tuned on the natural LIMA, respectively. These findings strongly demonstrate our key findings: unnatural languages are not bugs but features for LLMs. In addition, we attempt to understand the mechanisms by which LLMs process such unnatural languages. We demonstrate that LLMs process unnatural languages by effectively filtering out irrelevant tokens. Furthermore, LLMs combine relevant tokens from unnatural languages and infer contextual meaning in response to natural version questions. 2 Unnatural Languages Searching

Read the complete text