From Word2Vec to Transformers: Text-Derived Composition Embeddings for Filtering Combinatorial Electrocatalysts

Lei Zhang, Markus Stricker · 2026

Read From Word2Vec to Transformers: Text-Derived Composition Embeddings for Filtering Combinatorial Electrocatalysts by Lei Zhang, Markus Stricker (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-08881.txt.

Introduction

Lei Zhang and Markus Stricker screen catalyst mixes with text-derived codes, no lab labels needed. Each mix is coded from science texts, then ranked by likeness to conductivity and dielectric ideas in a two-axis map. On fifteen test sets, the simple Word2Vec baseline often cuts the pool most while staying near the best scores.

Catalog record

CREATORLei Zhang, Markus Stricker
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE41997
LICENSEhttp://creativecommons.org/licenses/by-sa/4.0/
RIGHTSCC BY SA
SOURCEarXiv

Source: https://arxiv.org/abs/2603.08881

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

From Word2Vec to Transformers: Text-Derived Composition Embeddings for Filtering Combinatorial Electrocatalysts Lei Zhang, Markus Stricker Abstract Compositionally complex solid solution electrocatalysts span vast composition spaces, and even one materials system can contain more candidate compositions than can be measured exhaustively. Here we evaluate a label-free screening strategy that represents each composition using embeddings derived from scientific texts and prioritizes candidates based on similarity to two property concepts. We compare a corpus-trained Word2Vec baseline with transformer-based embeddings, where compositions are encoded either by linear element-wise mixing or by short composition prompts. Similarities to ‘concept directions’, the terms conductivity and dielectric, define a 2-dimensional descriptor space, and a symmetric Pareto-front selection is used to filter candidate subsets without using electrochemical labels. Performance is assessed on 15 materials libraries including noble metal alloys and multicomponent oxides. In this setting, the lightweight Word2Vec baseline, which uses a simple linear combination of element embeddings, often achieves the highest number of reductions of possible candidate compositions while staying close to the best measured performance. I Introduction Compositionally complex solid solution materials, including high-entropy alloys and multicomponent oxides, provide a large compositional design space for electrocatalyst discovery because small changes in elemental identity and ratio can shift electronic structure, surface chemistry, and charge transport 12 . Combinatorial synthesis and high-throughput electrochemical measurements enable comparatively rapid exploration of this space 13 , but even a single library can contain hundreds to thousands of distinct compositions prepared under specific processing conditions 26 ; 1 . As library size and chemical diversity increase, deciding which compositions merit further measurement or deeper characterisation becomes a central practical step in experimental workflows. Screening strategies in this setting must balance two needs that pull in different directions: reducing the number of candidates enough to save experimental effort while still keeping at least one near-top performer for a given material system and operating condition. Supervised learning can be effective when labelled data are abundant, consistent, and comparable across libraries 17 ; 11 , yet in practice measurements are often sparse, material system-specific, and influenced by preparation details. This makes it attractive to consider approaches that can operate with weak or no labels and that reuse information already present in the scientific record. Scientific texts provide a compact representation of community knowledge linking composition to property language that is not a direct proxy for electrochemical response but often reflects underlying physical constraints and correlations 25 ; 14 . The materials and electrocatalysis literature repeatedly connects composition to descriptors such as electrical conductivity, dielectric behavior, oxidation tendencies, and other recurring concepts 24 ; 3 ; 20 . If compositions can be embedded into a latent space derived from text, then similarity to selected concept directions can define a low-dimensional descriptor space for ranking and filtering candidates in a way that is transferable across different libraries 28 . At the same time, a text-guided filter depends sensitively on how a multicomponent composition is represented within the embedding model 21 ; 8 . One option is to represent a composition as a concentration-weighted mixture of element embeddings, which is computationally straightforward and naturally supports fast screening, but it assumes linearity and independence across elements 19 . Another option is to embed a formatted full-composition prompt as a single piece of text, allowing the encoder to represent stoichiometric ratios and multi-element motifs more directly, but introducing choices about prompt design and model behavior 10 . Different embedding backbones further impose different inductive biases: lightweight distributional models trained on a focused corpus emphasise co-occurrence structure 9 , while transformer encoders may inject broader contextual associations learned during pretraining 4 . These modelling decisions shape both how aggressively candidate compositions in a given materials space, e.g. a materials library, can be reduced and how reliable high-performing compositions remain in the retained subset that comprises ‘the prediction’. In this work, we construct text-derived composition embeddings using Word2Vec 15 ; 16 ; 6 , MatSciBERT 7 , and Qwen 2 in both element-wise and full-composition prompt forms, project each composition onto concept directions related to electrical conductivity and dielectric behavior, and apply a dual Pareto-front filter to select candidates. We evaluate the resulting trade-offs between best-current retention and candidate reduction across 15 combinatorial libraries spanning the hydrogen evolution reaction (HER), oxygen reduction reaction (ORR), and oxygen evolution reaction (OER). II Methods II.1 Corpus collection (Word2Vec baseline) For the Word2Vec baseline, scientific abstracts related to electrocatalysis, high-entropy alloys and complex oxides are collected using the PaperCollector module of the MatNexus framework 27 . Abstracts are sourced from Scopus and arXiv, restricted to open-access publications available up to the year 2024. Only English-language abstracts are included. Metadata and full-text abstracts are automatically retrieved and stored in structured CSV format. This corpus is used exclusively to train the Word2Vec model; the transformer-based models (MatSciBERT and Qwen variants) are used in inference-only mode without additional training or fine-tuning on this corpus. II.2 Text preprocessing (Word2Vec baseline) Raw abstracts are processed using the TextProcessor module in MatNexus . Preprocessing includes removal of licensing statements and publisher-specific fragments (for example, “©” strings or publisher boilerplate), standard stopword filtering and sentence segmentation. Chemical element symbols and formulas are explicitly preserved to maintain their semantic relevance for materials-related embeddings. The cleaned, tokenised abstracts form the input for training the Word2Vec model. Transformer-based embeddings operate directly as pre-trained encoders and do not require this corpus-level preprocessing. II.3 Construction of text-derived composition embeddings Each composition in the experimental materials libraries is represented as a vector in a latent “text space”, and all downstream analysis (similarity to concepts and Pareto filtering) is performed on these vectors. The models differ only in how this composition embedding is constructed; the similarity and Pareto procedures are identical across all methods. II.3.1 Word2Vec baseline (W2V) As a lightweight baseline, we train word embeddings using the skip-gram architecture implemented in the VecGenerator module of MatNexus , based on the Word2Vec model 15 ; 16 ; 6 . Compared with TF-IDF or large transformer models, Word2Vec provides compact, dense vector representations at comparatively low computational cost, making it suitable for focused scientific corpora 22 . Training is performed with 200-dimensional vectors, a window size of 5, hierarchical softmax and a minimum word frequency of 1. All available CPU threads are used for parallelisation. Element symbols (for example, “Ag”, “Pd”, “Pt”) are treated as ordinary tokens in the corpus. For a given binary or multicomponent composition c , we first extract the normalised atomic fractions of all present elements from the materials library (for example, Ag 0.5 Pd 0.5 ). The corresponding composition vector \mathbf{v}_{\text{W2V}}(c) is computed as a concentration-weighted linear combination of the element embeddings, \mathbf{v}_{\text{W2V}}(c)\;=\;\sum_{i}x_{i}\,\mathbf{w}_{i}, where x_{i} is the atomic fraction of element i and \mathbf{w}_{i} is its Word2Vec embedding. The resulting vector is \ell_{2} -normalised prior to similarity calculation. An example of the composition embeddings distribution is shown in Fig. a). II.3.2 Element-wise transformer embeddings (MatSciBERT and Qwen) To probe the benefit of contextual transformer embeddings while retaining the simple linear-combination scheme, we construct element-wise material vectors using two models: • MatSciBERT 7 : a domain-specific transformer model applied to short element sentences. • Qwen 2 : a Qwen-based embedding model accessed via the Blablador service 23 ,

Read the complete text