Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces
Zakaria Elabid, Jan Andrzejewski, Bartosz Brzoza, Attila Cangi · 2026
Read Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces by Zakaria Elabid, Jan Andrzejewski, Bartosz Brzoza, Attila Cangi (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06303.txt.
Introduction
Zakaria Elabid and three co-authors test whether steering directions in a molecule generating model's latent space reflect real chemistry. Generative models assume their latent geometry is meaningful, but apparent property patterns can come from shortcuts in the sequence encoding instead. The team study an unsupervised transformer variational model trained on SELFIES, a text encoding of molecules, and then freeze it for analysis.
Linear probes fitted to chemical inputs supply possible steering directions. A confound aware check splits true chemical signal from encoding artifacts like chain length and token counts, which the latent space encodes strongly. Six properties, including lipophilicity and polar surface area, showed robust steady steering under that scrutiny. Other properties were better described by local latent gradients than by one global direction.
Catalog record
| CREATOR | Zakaria Elabid, Jan Andrzejewski, Bartosz Brzoza, Attila Cangi |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 100 KB |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.06303
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Molecules Meet Language: Confound-Aware Representation Learning and Chemical Property Steering in Transformer-VAE Latent Spaces Zakaria Elabid, Jan Andrzejewski, Bartosz Brzoza, Attila Cangi Abstract Molecular generative models often assume meaningful latent geometry, but apparent property predictability can reflect sequence-level shortcuts rather than chemical organization. We study this issue in an unsupervised autoregressive Transformer-VAE trained on SELFIES. After training, we freeze the model, fit linear probes to RDKit descriptors, and use the probe weights as candidate global steering directions. To separate chemical signal from SELFIES artifacts, we introduce a confound-aware evaluation based on residualization, confound-direction alignment analysis, and decoded-molecule traversal. This is necessary because SELFIES length, branch tokens, ring tokens, and token entropy are strongly encoded in the latent space. Under this confound-aware evaluation, we find robust monotonic steering for cLogP, FractionCSP3, HeavyAtomCount, TPSA, BertzCT, and HBA. Nonlinear probes further show that some properties admit stable global directions, while others are better described by local latent gradients. Overall, our results show that chemically meaningful steering can emerge in entangled molecular latent spaces, but only when validated through decoded molecules and controlled for representation-level confounds. 1 Introduction Molecular design is a central problem in drug discovery and materials science, where the objective is to generate or optimize chemical structures with desired properties [ 22 , 23 , 9 ] . Although molecules are naturally represented as labeled graphs, molecular line notations make them accessible to sequence models. SMILES provides a compact chemical language for molecular graphs [ 27 ] , and its non-unique forms have been used as data augmentation in molecular learning [ 5 ] . SELFIES addresses a different but crucial issue for generation: arbitrary SELFIES strings decode to valid molecules under the representation grammar [ 20 ] . In this work, molecular language is therefore used as the training interface, while chemical validity and molecular properties are evaluated after decoding into graph-based molecular objects. Language modeling has become a general mechanism for learning representations from unlabeled sequences. Transformers enabled scalable sequence modeling through attention [ 24 ] , while large-scale pretraining demonstrated that useful representations can emerge without task-specific supervision [ 8 ] . Related principles have been observed in biological sequences [ 21 ] and chemical strings, including SMILES Transformer fingerprints [ 16 ] , ChemBERTa-style molecular pretraining [ 6 ] , SELFIES-based Transformer representations [ 29 ] , and autoregressive molecular generation with MolGPT [ 3 ] . These works support the view that molecular strings contain learnable chemical regularities. However, they do not fully answer whether an unsupervised molecular language model learns a continuous latent space with chemically meaningful and steerable directions. Molecular generation and optimization have followed complementary directions. Optimization searches in discrete molecular graph or string space using reinforcement learning, genetic search, or graph-based policies [ 28 , 31 , 17 , 12 ] . Generative approaches instead learn a distribution over molecules and use the learned representation for sampling, interpolation, or optimization, including molecular VAEs, junction-tree models, normalizing flows, and graph diffusion models [ 13 , 18 , 30 , 25 ] . Latent-space optimization exploits trained generators as continuous search spaces, for example through Bayesian optimization, latent inceptionism, linear molecular manipulation, or learned latent flows [ 14 , 11 , 10 , 26 ] . These methods show that latent molecular spaces can support optimization, in a nontrivial manner: a useful generator does not necessarily provide globally meaningful chemical directions. This paper studies that question in a deliberately constrained setting. We train a slot-based autoregressive Transformer-VAE on SELFIES sequences using only reconstruction and latent regularization. After training, the encoder and decoder are frozen. We then fit linear probes to RDKit-computed molecular descriptors and use the learned probe weights as candidate global steering directions. Thus, property labels are not used to train the generative model; they are used only post hoc to interrogate the latent space. The main evaluation is not whether a descriptor can be predicted from a molecular string, but whether moving through the frozen latent space along a probe-derived direction changes decoded molecules in the intended chemical direction. A central risk is that a molecular language model may encode sequence-level shortcuts rather than chemical structure. SELFIES improves validity, but it remains a token sequence whose length, branch tokens, ring tokens, and token entropy can correlate with molecular size, topology, and descriptor values. We therefore evaluate latent directions under a confound-aware protocol: descriptors must be predictable from the frozen latent representation, robust to measured SELFIES-level confounds, and validated after decoding through molecular properties rather than latent probe scores alone. The full framework is summarized in Figure 1 . Our contributions are as follows: • We frame molecular property steering as post hoc interpretation of an unsupervised molecular language model trained on SELFIES. • We connect linear probing to latent chemical control by treating probe weights as global steering axes and validating them through decoded molecules. • We introduce a confound-aware evaluation that separates chemical organization from SELFIES-level shortcuts such as token length, branch-token count, ring-token count, and token entropy. • We show that useful global directions can emerge in an entangled latent space, with cLogP, FractionCSP3, BertzCT, TPSA, and HBA exhibiting robust monotonic steering. • We use nonlinear probes diagnostically to distinguish globally linear steerability from local or nonlinear latent organization. Figure 1 : Overview of the proposed framework. A. SMILES are converted to SELFIES, tokenized, and used to train an autoregressive Transformer-VAE. B. The encoder is frozen and the latent space is probed for molecular properties Y and SELFIES-level confounds C ; residualization removes the component predictable from the confounds. C. Raw and residual R^{2} values, together with decoded-molecule traversals, separate linearly steerable properties from confounded or nonlinear ones. 2 Background 2.1 Molecular representations and chemical language Molecules are graphs whose atoms and bonds define chemical structure. Graph-based generators model this structure directly, avoiding arbitrary traversal order, and have been developed through junction-tree decomposition, implicit graph generation, invertible graph flows, and discrete graph diffusion [ 18 , 7 , 30 , 25 ] . Their advantage is structural explicitness; their difficulty is that valid molecular generation remains a constrained discrete problem involving rings, graph size, and graph isomorphism. String representations expose the same chemical objects to sequence models. SMILES is compact, expressive, and widely supported, but valid generation requires learning a constrained formal language involving branches, ring closures, aromaticity, and stereochemical conventions [ 27 ] . Canonicalization can impose a deterministic form, while non-canonical SMILES can also be exploited as augmentation [ 5 ] . SELFIES changes this trade-off by embedding stronger validity constraints into the representation grammar, making it particularly suitable for sequence-based generative modeling [ 20 ] . 2.2 Chemical language models and latent molecular generation Transformers provide a natural architecture for molecular strings because autoregressive decoding can model token sequences one symbol at a time [ 24 ] . More broadly, self-supervised language modeling has shown that useful structure can be recovered from unlabeled sequences in natural language and biology [ 8 , 21 ] . Chemical language models extend this principle to molecular strings through unsupervised SMILES pretraining, large-scale molecular Transformers, SELFIES-based representation learning, and GPT-style molecular generation [ 16 , 6 , 29 , 3 ]