SMOTExT: SMOTE meets Large Language Models
Mateusz Bystroński, Mikołaj Hołysz, Grzegorz Piotrowski, Nitesh V. Chawla, Tomasz Kajdanowicz · 2025
Read SMOTExT: SMOTE meets Large Language Models by Mateusz Bystroński, Mikołaj Hołysz, Grzegorz Piotrowski, Nitesh V. Chawla, Tomasz Kajdanowicz (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-13434.txt.
Introduction
Text data sets are often lopsided, with rare classes starved of examples for training. Mateusz Bystroński and co-authors adapt a classic fix, oversampling the minority, to raw text in a 2025 paper using language models as the printer.
The method blends the numeric fingerprints of two real examples, then decodes the in-between point back into fresh text with a retrieval-generation system. Early results rest on qualitative inspection rather than full benchmarks, which the authors state plainly. Two uses stand out: sharpening models in few-example settings, and privacy. Models trained only on the synthetic text performed about as well as those trained on the originals, a promising sign for learning from data that cannot be shared.
Catalog record
| CREATOR | Mateusz Bystroński, Mikołaj Hołysz, Grzegorz Piotrowski, Nitesh V. Chawla, Tomasz Kajdanowicz |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 22 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-13434
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SMOTExT: SMOTE meets Large Language Models Mateusz Bystroński, Mikołaj Hołysz, Grzegorz Piotrowski, Nitesh V. Chawla, Tomasz Kajdanowicz Abstract Data scarcity and class imbalance are persistent challenges in training robust NLP models, especially in specialized domains or low-resource settings. We propose a novel technique, SMOTExT , that adapts the idea of Synthetic Minority Over-sampling (SMOTE) to textual data. Our method generates new synthetic examples by interpolating between BERT-based embeddings of two existing examples and then decoding the resulting latent point into text with xRAG architecture. By leveraging xRAG’s cross-modal retrieval-generation framework, we can effectively turn interpolated vectors into coherent text. While this is preliminary work supported by qualitative outputs only, the method shows strong potential for knowledge distillation and data augmentation in few-shot settings. Notably, our approach also shows promise for privacy-preserving machine learning: in early experiments, training models solely on generated data achieved comparable performance to models trained on the original dataset. This suggests a viable path toward safe and effective learning under data protection constraints. 1 Introduction Building reliable NLP models in low-resource scenarios or for under-represented classes remains a major challenge due to the limited availability of annotated training data and ethical concerns. Data augmentation offers a promising strategy to mitigate this issue by generating additional synthetic examples. However, common textual augmentation techniques—such as synonym replacement, random deletion, or back-translation—often introduce semantic drift, grammatical errors, or distributional inconsistencies. In contrast, interpolation-based augmentation has proven effective in continuous feature spaces, especially in domains like vision or tabular data. The S ynthetic M inority O ver-sampling TE chnique (SMOTE) Chawla et al. 2002 is a well-known example that creates new minority-class samples by linearly interpolating between feature vectors. Adapting such interpolation to text, however, is non-trivial due to the discrete and high-dimensional nature of natural language. Figure 1: Two textual examples A and B are encoded into embeddings. An interpolated vector SMOTExT is fed into the xRAG model via its modality bridge (projector). Language model then generates a synthetic text that reflects combined features of the inputs. We propose a novel method— SMOTExT —that performs SMOTE-style interpolation for text by operating in a continuous latent space. Specifically, we embed two textual examples using a pre-trained encoder and compute a linear interpolation between their embeddings. This intermediate vector is then decoded into natural language using xRAG Cheng et al. 2024 . This approach enables the generation of realistic, domain-consistent samples that can enrich training data in few-shot or imbalanced-class settings. Compared to perturbation-based augmentation, interpolation in latent space may better preserve semantic relationships between examples. However, interpolated vectors are not guaranteed to remain on the natural language manifold, and out-of-distribution artifacts may occur—particularly when inputs differ significantly. Nevertheless, by decoding these vectors using a strong generative model, we can often obtain fluent and contextually appropriate text, especially when source examples are from the same domain or class. Our contributions are as follows: (1) We introduce a novel textual data augmentation technique based on embedding space interpolation, inspired by SMOTE and enabled by large-scale pre-trained models; (2) We provide a real-world proof-of-concept in the medical domain, showing that our method can synthesize medically plausible patient descriptions; (3) We conduct a preliminary quantitative evaluation on a general-purpose classification task, showing that the method improves metrics in low-resource settings and approximates real data distribution when used alone. and (4) We discuss limitations of the current approach and outline directions for future quantitative evaluation, including applications in privacy-preserving and generative NLP tasks. 2 Related Work Text Data Augmentation. Numerous augmentation methods address data scarcity in NLP. Lexical approaches like EDA Wei and Zou 2019 apply random synonym replacement, insertion, or deletion, but often risk semantic drift or ungrammaticality. Paraphrasing via back-translation Sennrich et al. 2016 or prompt-based LLM generation offers more fluency, though may drift off-distribution without constraints. Other techniques operate in continuous representation spaces. Mixup Zhang et al. 2018 ; Verma et al. 2019 , originally for vision, has been adapted to NLP by interpolating hidden states. TMix Chen et al. 2020 and SSMix Yoon et al. 2021 create intermediate representations by mixing hidden layers in transformer models, improving training. However, these methods are typically tied to specific architectures and do not generate standalone text. In contrast, our method decouples augmentation from model training by decoding interpolated embeddings into human-readable text. This enables model-agnostic data augmentation and facilitates qualitative inspection of synthetic examples. SMOTE and Interpolation. Our approach is inspired by SMOTE Chawla et al. 2002 , a classic oversampling technique that mitigates class imbalance by interpolating between a minority class instance and one of its neighbors in feature space. This generates synthetic samples within the minority region, avoiding duplication. While effective in continuous domains (e.g., numeric or image data), applying SMOTE to text requires a suitable latent representation. DeepSMOTE Dablain et al. 2023 extended this idea to high-dimensional data by performing interpolation in the latent space of a neural encoder. It maps images into a semantic embedding space, applies SMOTE-style interpolation, and reconstructs synthetic examples via a decoder—circumventing unrealistic blending in pixel space. We adopt a similar principle for text, operating directly in the embedding space of pre-trained language models. xRAG. The recently proposed xRAG model Cheng et al. 2024 enables language models to condition on a single dense vector, derived from a retrieved document, via a special token. A fixed retriever generates document embeddings, which are projected into the LM’s input space using a small trainable modality bridge . Only this projection is trained, keeping both the retriever and the LM frozen. The LM attends to the resulting embedding and generates fluent, context-aware output. 3 Proposed Method Our approach leverages xRAG’s capability to integrate and decode an arbitrary fixed-length embedding into natural language. Essentially, we repurpose the xRAG architecture as a decoder for interpolated example embeddings. Prior work Cai et al. 2021 has shown that BERT Devlin et al. 2019 embeddings tend to form well-structured and concentrated clusters in the latent space, making linear interpolation between them a viable strategy for generating semantically meaningful intermediate representations. This structural property supports the idea that interpolated points in embedding space can correspond to plausible language outputs. xRAG provides an off-the-shelf, pre-trained mechanism to realize such vectors as text, without the need to train a new autoencoder or decoder architecture from scratch. By treating the interpolated vector as if it were a retrieved document embedding, we enable the language model to generate a coherent sentence that blends features of the source examples. Our method, consists of three main steps: (1) obtaining vector representations of input texts; (2) interpolating between two such vectors to produce a new latent point; and (3) decoding this latent vector into text using xRAG. Embedding Text Examples. We first convert each text example into a fixed-dimensional embedding. In this work we use the [CLS] representation from SRF-Embedding-Mistral Meng et al. 2024 . Let x_{i} and x_{j} be two text examples (e.g., two sentences or short paragraphs). We obtain e_{i}=\text{BERT-Enc}(x_{i}) and e_{j}=\text{BERT-Enc}(x_{j}) , where e_{i}\in\mathbb{R}^{d} (with d=4096 in our case). These embeddings are intended to capture the major features of each text in a continuous vector form. Latent Space Interpolation. Next, we generate a new latent point between e_{i} and e_{j} .