Unlocking Chemical Insights: Superior Molecular Representations from Intermediate Encoder Layers
Luis Pinto · 2025
Read Unlocking Chemical Insights: Superior Molecular Representations from Intermediate Encoder Layers by Luis Pinto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-06443.txt.
Introduction
Luis Pinto shows that the middle layers of pre-trained molecule encoders hold richer information than the final layer everyone uses, in a 2025 paper. Encoders that turn molecules into number vectors feed tasks like property prediction, and standard practice reads only the last layer's output. Pinto first traced how information flows through five different encoders.
Middle layers kept broad, general features, while the final layer squeezed information into a narrower shape. Across 22 prediction tasks, frozen middle-layer vectors beat final-layer ones by about five percent on average, and sometimes by far more. Fine-tuning a shortened encoder did even better, gaining about nine percent on average and setting new records on several benchmarks.
Catalog record
| CREATOR | Luis Pinto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 48 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2506-06443
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Unlocking Chemical Insights: Superior Molecular Representations from Intermediate Encoder Layers Luis Pinto Abstract Pretrained molecular encoders have become indispensable in computational chemistry for tasks such as property prediction and molecular generation. However, the standard practice of relying solely on final-layer embeddings for downstream tasks may discard valuable information. In this work, we challenge this convention by conducting a comprehensive layer-wise analysis of five diverse molecular encoders across 22 ADMET property prediction tasks. Our results demonstrate that embeddings from intermediate layers consistently outperform final-layer representations. Specifically, using fixed embeddings from the optimal intermediate layers improved downstream performance by an average of 5.4%, reaching gains up to 28.6%. Furthermore, finetuning up to these intermediate layers yielded even greater average improvements of 8.5%, with performance increases as high as 40.8%, achieving new state-of-the-art results on several benchmarks. Additionally, a strong positive correlation between fixed embedding performance and finetuning outcomes supports an efficient evaluate-then-finetune approach, enabling identification of optimal layers with reduced computational cost. These findings highlight the importance of exploring the full representational depth of molecular encoders to achieve substantial performance improvements and computational efficiency. The code is made publicly available at https://github.com/luispintoc/Unlocking-Chemical-Insights . 1 Introduction Deep learning has reshaped molecular science, where pretrained molecular encoders are essential tools for applications from virtual screening in drug discovery to designing novel materials with desired properties goh2018smiles2vecinterpretablegeneralpurposedeep ; 10.1093/bioinformatics/bty593 ; Xu2023 ; sypetkowski2024on . These encoders, built on architectures such as Transformers vaswani2023attentionneed and Graph Neural Networks (GNNs) NIPS2017_5dd9db5e ; kipf2017semisupervisedclassificationgraphconvolutional , learn rich representations of molecular structures that can be applied to a wide range of downstream predictive tasks. The common practice of extracting molecular representations from the final encoder layer, while simple and widely adopted, rests on the implicit assumption that this layer consistently provides the most informative and task-relevant features. However, this assumption is increasingly challenged by findings from Natural Language Processing (NLP) and Computer Vision, where intermediate layers often encode richer, more generalizable, or task-specific information, leading to significant performance gains liu-etal-2019-linguistic ; jin-etal-2025-exploring ; gurnee2024languagemodelsrepresentspace ; valeriani2023geometryhiddenrepresentationslarge . Despite the unique data modalities (e.g. molecular graphs, 3D conformers) and distinct pretraining objectives inherent to cheminformatics, systematic layer-wise investigations of molecular encoders have been notably scarce. This leaves a critical question unanswered: are we overlooking superior molecular representations by adhering to the final-layer convention in chemistry? Figure 1: Percentage improvement in test metric of the best performing intermediate layer relative to the final-layer when evaluating fixed embeddings. A "–" symbol indicates that the final-layer was better. To address this, we propose and implement a comprehensive two-stage evaluation study. First, we assess the quality of fixed embeddings extracted from every layer of various pretrained molecular encoders. Second, we evaluate the performance when the encoder is finetuned up to each respective layer for specific downstream tasks. This dual approach allows us to understand both the intrinsic quality of layer-wise representations and their potential as effective priors for task-specific adaptation. Our main contributions are as follows: 1. We conduct an empirical study evaluating fixed embeddings from five diverse molecular encoders across 22 absorption, distribution, metabolism, excretion, and toxicity (ADMET) tasks. We demonstrate that in over 81% of model-task combinations, an intermediate layer outperforms the conventionally used final-layer, yielding an average increase of 5.4% in downstream performance. 2. We further show that these advantages persist when the encoders are finetuned up to the corresponding intermediate layers for the same set of tasks, again observing that in over 71% of cases, intermediate layers lead to notable improvements over finetuning the complete encoder, with an average gain of 8.5%. 3. We establish a strong positive Pearson correlation between the performance of fixed embeddings and finetuning performance on a per-layer basis. This suggests a practical evaluate-then-finetune strategy: fixed embedding performance can serve as a reliable guide for selecting the best layer to finetune, reducing both computational cost and trial-and-error. 2 Related Work 2.1 Molecular Encoders Molecular encoders are deep learning models designed to transform chemical structures into fixed-length numerical vectors, also known as embeddings, that can be used for downstream tasks such as property prediction, molecular generation, or virtual screening. These models play a central role in computational chemistry and drug discovery by enabling machine learning algorithms to efficiently capture the relevant chemical, structural, and physical information inherent to molecules. The effectiveness of a molecular encoder is critically dependent on both the representational richness of its learned embeddings and its ability to generalize across diverse chemical spaces. Molecular encoders are generally differentiated by two main aspects: the type of molecular input they process, for example SMILES doi:10.1021/ci00057a005 strings, molecular graphs, or 3D conformers; and the underlying computational architecture, which could be Transformers, Graph Neural Networks (GNNs), or other specialized architectures. Transformer-based sequence models such as ChemBERTa chithrananda2020chembertalargescaleselfsupervisedpretraining , ChemGPT frey_soklaski_axelrod_samsi_gomez-bombarelli_coley_gadepally_2022 , and MolFormer Ross2022 are pretrained on textual representations of molecules and apply masked language modeling devlin-etal-2019-bert or autoregressive objectives Radford2019LanguageMA to learn general purpose embeddings. ChemBERTa and MolFormer operate on tokenized SMILES strings, whereas ChemGPT is trained on SELFIES Krenn_2020 , a robust string grammar that guarantees valid molecules. These models inherit the architectural benefits of NLP transformers, including multi-head self-attention and position embeddings. Hybrid 2D/3D transformer encoders like Uni-Mol zhou2023unimol extend transformer architectures to directly incorporate both 2D topological information and 3D atomic coordinates. Uni-Mol leverages pairwise distance matrices and spatial positional encodings to inject geometric priors, enabling the model to learn both conformation-aware and topology-sensitive representations. Its architecture retains the global receptive field of transformers while being sensitive to the spatial layout of molecules. Graph Neural Networks for 2D molecular graphs represent a distinct branch of molecular representation learning. These models operate directly on the 2D graph structure of molecules, where atoms are nodes and bonds are edges. Through iterative message passing steps, GNNs aggregate information from neighboring atoms and bonds to learn representations that capture molecular topology and chemical features. Models like the Directed Message Passing Neural Network (D-MPNN) HAN2022100201 are often trained with supervised learning on specific molecular properties. Others, such as MolCLR Wang2022 , employ self-supervised learning strategies, like contrastive learning between positive and negative molecular pairs, to learn general purpose embeddings. GNN-based 3D models employ graph neural networks to process three dimensional molecular structures. Some of these models incorporate equivariant architectures to inherently respect physical symmetries, while others may utilize data augmentation techniques to learn these invariances. A specific subset of these 3D GNN models includes machine learning force fields Behler2016-og ; doi:10.1021/acs.chemrev.0c01111 , which are trained to predict molecular energies and forces from DFT calculations PhysRev.140.A1133 . For instance, MACE Batatia2022mace and Pos-EGNN Pos-EGNN are examples that use an equivariant architecture to predict such properties. In contrast, Orb models rhodes2025orbv3atomisticsimulationscale do not rely on equivariant architectures but instead leverage augmentations. 2.2 Layer-Wise Probing in Language and Vision Models Most relevant to our study is the recent work of Skean et al. (2025) skean2025layerlayeruncoveringhidden , who performed the most comprehensive analysis to date of intermediate-layer representations in deep neural networks. Their layer-by-layer investigation across transformer and state-space models in both language and vision domains revealed that intermediate layers often exhibit higher utility than final ones across a range of probing tasks. Using a unified framework integrating metrics from information theory, geometry,