Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences

Artem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev Andrey Savchenko, Maksim Makarenko · 2026

Read Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences by Artem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev Andrey Savchenko, Maksim Makarenko (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-15713.txt.

Introduction

Sakhno and co-authors bridge a gap between learned embeddings and handcrafted features. Finance systems run on event sequences like transactions and user actions. Shop-floor pipelines still trust hand-built stats for clarity and speed. Learned embeddings hold rich signals but stay hard to read. Their EAFD framework pairs pretrained embeddings with an LLM feature agent. The agent proposes features, then scores each one twice. Alignment checks what embeddings already hold. Complementarity hunts signals they miss. Gains reached 5.8 percent over top embeddings.

Catalog record

CREATORArtem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev Andrey Savchenko, Maksim Makarenko
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE32397
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.15713

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Embedding-Aware Feature Discovery: Bridging Latent Representations and Interpretable Features in Event Sequences Artem Sakhno, Ivan Sergeev, Alexey Shestov, Omar Zoloev, Elizaveta Kovtun, Gleb Gusev Andrey Savchenko, Maksim Makarenko Abstract Industrial financial systems operate on temporal event sequences such as transactions, user actions, and system logs. While recent research emphasizes representation learning and large language models, production systems continue to rely heavily on handcrafted statistical features due to their interpretability, robustness under limited supervision, and strict latency constraints. This creates a persistent disconnect between learned embeddings and feature-based pipelines. We introduce Embedding-Aware Feature Discovery (EAFD), a unified framework that bridges this gap by coupling pretrained event-sequence embeddings with a self-reflective LLM-driven feature generation agent. EAFD iteratively discovers, evaluates, and refines features directly from raw event sequences using two complementary criteria: alignment , which explains information already encoded in embeddings, and complementarity , which identifies predictive signals missing from them. Across both open-source and industrial transaction benchmarks, EAFD consistently outperforms embedding-only and feature-based baselines, achieving relative gains of up to + 5.8 % +5.8\% over state-of-the-art pretrained embeddings, resulting in new state-of-the-art performance across event-sequence datasets. 1 Introduction Industrial machine learning systems commonly operate on temporal event sequences, including financial transactions 28 ; 9 , online purchase histories 25 , electronic health records 8 , spatial trajectories 44 . In production environments, these sequences are processed under strict constraints on latency, throughput, and stability 42 , which has led to the widespread adoption of specialized encoder models that produce compact embeddings 6 , alongside handcrafted statistical interpretable features that encode domain-specific structure 41 ; 36 . While recent research has increasingly focused on representation learning 37 , real-world systems continue to rely heavily on feature-based pipelines 26 ; 10 due to their ease of configuration for new tasks with limited labeled data, interpretability, and predictable runtime behavior. Figure 1: Blind spots of embeddings. Coefficient of determination ( R^{2} ) of a feature reconstruction based on embedding representations CoLES, NTP, and LLM4ES on the Rosbank dataset. Values highlight systematic representational blind spots across embeddings. Recent advances in automated feature engineering 7 and AutoML 11 have greatly reduced the manual effort required to construct and select features for structured data. Many end-to-end AutoML frameworks incorporate feature selection directly into the training pipeline 2 , while other approaches focus explicitly on automated feature construction. For example, AutoFeat 17 generates non-linear transformations for tabular data, and Deep Feature Synthesis (DFS), implemented in Featuretools 18 , produces fixed-size representations from sequential inputs using predefined aggregation operators. More recently, research in automated feature engineering has drifted toward the use of large language models (LLMs) as reasoning and orchestration components, moving toward fully automated and increasingly agentic frameworks that coordinate feature generation, selection, and evaluation 16 ; 13 ; 1 . Despite this progress, automated feature engineering for sequential data largely remains decoupled from representation learning. As a result, deep embeddings and statistical features are optimized through largely separate workflows. The consequences of this separation become apparent when examining how well embeddings reconstruct different classes of handcrafted features (Figure 1 ). While embeddings capture certain aggregate statistics with high fidelity, other feature groups exhibit persistently low reconstruction quality, revealing systematic representational blind spots that vary across embedding models. This is problematic in practice: information missing from the embedding cannot be recovered by subsequent models, leading to performance ceilings and redundant feature construction. In the absence of a clear interface between representation learning and feature discovery, feature engineering remains unaware of representation gaps, and embedding refinement proceeds without interpretable guidance from structured signals. To address this gap: • We introduce Embedding-Aware Feature Discovery (EAFD) 1 1 1 The source code is provided in the supplementary materials. , a unified framework that transforms the disconnect between embeddings and structured features into an iterative, self-reflective reasoning process for feature discovery (Section 3 ). EAFD operates in two complementary regimes: an interpretability regime , which analyzes and explains the information already encoded in pretrained embeddings by aligning with interpretable features, and a performance regime , which discovers complementary features that provide additional predictive signal not present in embeddings. EAFD anchors on pretrained embeddings and integrates them with an LLM-driven feature-generation agent that actively explores raw event-sequence data, proposing and evaluating candidate features based on alignment and complementarity criteria. • We evaluate EAFD on four open-source event-sequence benchmarks, where it consistently outperforms embedding-only and feature-based baselines across datasets and backbone embeddings (Section 4.2 ). EAFD achieves relative gains of up to +5.8\% over state-of-the-art embeddings and up to +19\% over weaker representations, establishing a new state of the art on open-source transaction datasets. We further validate EAFD on a large-scale, proprietary, multitarget financial dataset, where a single EAFD-enhanced representation improves across classification and regression targets, yielding gains of up to 12.55\% and up to 3.87\% error reduction, respectively. • We use EAFD’s explicit feedback to analyze multiple state-of-the-art embedding models, uncovering systematic representational biases and information gaps (Section 4.3 ). Leveraging these insights, we propose several modifications for the CoLES framework, resulting in up to 1.20% relative improvement in churn prediction. We also demonstrate how EAFD’s embedding interpretability signals can be further coupled with a privacy-preserving feature erasure mechanism 35 ; 24 that identifies and suppresses sensitive attributes encoded in embeddings. Overall, EAFD provides a unified and practical framework for analyzing and enhancing event-sequence embeddings through feature discovery, bridging representation learning and feature-based pipelines across both benchmark datasets and large-scale industrial systems. Figure 2: Overview of Embedding-Aware Feature Discovery (EAFD). (a) Latent embedding pipeline mapping event sequences to continuous user representations. (b) EAFD agent loop: an LLM-based generator proposes interpretable features from raw sequences, evaluated by embedding–feature alignment and downstream utility, with reflective updates guiding iteration. (c) Feature outcomes: aligned features recover information encoded in the embedding, while complementary features capture predictive factors missing from it. (d) Embedding–feature space decomposition into latent, interpretable, and blind-spot regions. (e) Deployment and refinement: discovered features improve downstream tasks and support targeted encoder refinement (e.g., coverage, robustness, privacy). 2 Related Works Automated feature engineering ( AutoFE ) 7 has been widely studied for tabular data. Classical systems range from LightAutoML’s data cleaning 40 to more complex approaches like Featuretools 18 and AutoCross 23 , which use predefined transformation grammars and cross-feature search. Neural methods, such as TabNet 2 , learn latent feature representations directly, while OpenFE 43 improves generation diversity and efficiency via a fast two-stage pruning. Recent progress in large language models has extended this line of work toward feature-generation agents. CAAFE 16 shows that LLMs can synthesize meaningful features and executable code from natural-language descriptions, while follow-up approaches explore evolutionary search (LLM-FE 1 ), program synthesis (DS-Agent 13 ), and table-structure reasoning 14 . Both classical AutoFE systems and LLM-based feature agents exhibit a common limitation in modern industrial pipelines that rely on pretrained encoders: feature construction is performed independently of the embedding space, without assessing redundancy or complementarity, and without explicit reasoning over temporal structure in event-sequence data. A complementary research direction investigates embedding probing 21 and representation analysis 39 . Techniques such as linear probes 4 , sparse decomposition 15 , and centered kernel alignment 27 provide insight into what pretrained models encode. These

Read the complete text