CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization DOI: XXXXXXX.XXXXXXX Conference: ACM Conference; August 9-13, 2026; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Search methodologies CCS: Computing methodologies Machi

Beicheng Xu, Keyao Ding, Wei Liu, Yupeng Lu, Bin Cui · 2026

Read CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization DOI: XXXXXXX.XXXXXXX Conference: ACM Conference; August 9-13, 2026; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Search methodologies CCS: Computing methodologies Machi by Beicheng Xu, Keyao Ding, Wei Liu, Yupeng Lu, Bin Cui (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-09851.txt.

Introduction

Feature craft is pivotal in auto learning but stays a choke for old methods working in rigid spaces without domain sense. Beicheng Xu and four co-authors note large language models offer a hopeful swap making open ops with meaning. Yet current methods focus on solo subtasks like feature making, short of free chains. Rarely coupled with downstream tuning, they yield greedy feature-then-tuning workflows missing strong links. Their 2026 paper presents CoFEH, a joint frame mixing language-model feature craft and Bayesian tuning for auto learning.

CoFEH uses a language-model feature tuner powered by Tree of Thought to explore chains. A Bayesian tuning module handles tuning, and a tuner picker mixes step types on the fly. A mutual tuning gear shares context between language model and Bayesian tuner for both-ways calls. Tests show CoFEH beating old and language-model rivals in solo feature craft and settings.

Catalog record

CREATORBeicheng Xu, Keyao Ding, Wei Liu, Yupeng Lu, Bin Cui
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE104 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-09851

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

CoFEH: LLM-driven Feature Engineering Empowered by Collaborative Bayesian Hyperparameter Optimization DOI: XXXXXXX.XXXXXXX Conference: ACM Conference; August 9-13, 2026; Woodstock, NY ISBN: 978-1-4503-XXXX-X/2018/06 CCS: Computing methodologies Search methodologies CCS: Computing methodologies Machine learning algorithms Beicheng Xu, Keyao Ding, Wei Liu, Yupeng Lu, Bin Cui Abstract . Feature Engineering (FE) is pivotal in automated machine learning (AutoML) but remains a bottleneck for traditional methods, which treat it as a black-box search, operating within rigid, predefined search spaces and lacking domain awareness. While Large Language Models (LLMs) offer a promising alternative by leveraging semantic reasoning to generate unbounded operators, existing methods fail to construct free-form FE pipelines, remaining confined to isolated subtasks such as feature generation. Most importantly, they are rarely optimized jointly with hyperparameter optimization (HPO) of the ML model, leading to greedy “FE-then-HPO” workflows that cannot capture strong FE–HPO interactions. In this paper, we present CoFEH, a collaborative framework that interleaves LLM-based FE and Bayesian HPO for robust end-to-end AutoML. CoFEH uses an LLM-driven FE optimizer powered by Tree of Thought (TOT) to explore flexible FE pipelines, a Bayesian optimization (BO) module to solve HPO, and a dynamic optimizer selector that realizes interleaved optimization by adaptively scheduling FE and HPO steps. Crucially, we introduce a mutual conditioning mechanism that shares context between LLM and BO, enabling mutually informed decisions. Experiments show that CoFEH not only outperforms traditional and LLM-based FE baselines, but also achieves superior end-to-end performance under joint optimization. 1. Introduction Figure 1 . Comparison of optimization workflows: existing methods vs. CoFEH. Optimization Pipeline of Existing Methods and \sys. The success of machine learning (ML) hinges on the synergy between data representation and model capacity ( 25 ) . Therefore, Feature Engineering (FE) serves as the cornerstone of an ML pipeline, directly influencing ML models’ performance and effectiveness. Broadly defined, FE constitutes a holistic transformation process that bridges raw data and model inputs, encompassing preprocessing (e.g., imputation, normalization), transformation (e.g., encoding, discretization), generation (synthesizing new interactions), and selection (filtering irrelevant signals) ( 39 ) . However, manual FE is labor-intensive and demands deep domain expertise. To lower barriers and streamline deployment, the AutoML community has proposed several methods to automate the FE process ( 52 ; 56 ; 19 ; 26 ; 27 ; 28 ) . Many end-to-end AutoML systems also incorporate FE by casting it as a search problem over a predefined space of transformation operations and pipeline templates ( 47 ; 11 ; 31 ; 30 ) . Leveraging an optimizer like Bayesian optimization (BO), they explore finite configurations to construct FE pipelines. Despite facilitating automation, these traditional approaches come with three fundamental drawbacks: (i) They are semantics-agnostic and depend on expensive trial-and-error without leveraging domain priors; (ii) They enforce rigid FE pipeline templates, often restricting pipelines to fixed sequences (e.g., generation followed by selection ( 19 ; 56 ) ) or isolated sub-tasks, precluding dynamic interleaving or flexible composition; (iii) They restrict the search to closed operation libraries (e.g., a small set of built-in preprocessing algorithms or basic arithmetic), which prevents discovering task-specific operations beyond predefined primitives. To address these gaps, Large Language Models (LLMs) have been introduced to automated FE ( 16 ; 12 ; 7 ; 50 ) . Through domain knowledge and code generation capabilities, LLMs can propose semantically grounded operations and bespoke logic beyond fixed operation libraries, alleviating drawbacks (i) and (iii) mentioned above. However, most existing methods target only a single FE subtask—most notably feature generation ( 18 ; 40 ; 12 ; 57 ; 1 ) —resulting in a homogeneous and simplistic FE pipeline. This leads to our first challenge: C1: Free-form FE. How can we empower LLMs to construct a truly free-form, human-expert-like FE pipeline that flexibly composes heterogeneous operations (preprocessing, transformation, generation, and selection) rather than only homogeneous ones? While LLMs are inherently well-suited for FE, existing methods overlook the critical synergy with Hyperparameter Optimization (HPO) of the downstream ML model. In modern AutoML, HPO is predominantly driven by Bayesian optimization (BO) due to its sample efficiency and uncertainty modeling ( 45 ; 43 ; 31 ) . Consequently, most LLM-based FE methods that aim for end-to-end performance must attach a BO-based HPO module. Because the two optimizers are heterogeneous and operate over differently represented spaces, they typically fall back to a greedy sequential routine—first optimizing FE under a fixed model, then tuning hyperparameters on the frozen features (Figure 1 , center) ( 18 ; 40 ; 12 ; 33 ) . This greedy strategy ignores the strong dependency between feature representation and model capacity. In contrast, traditional methods (e.g., Auto-sklearn ( 11 ) ) naturally enable joint optimization by enforcing a homogeneous search space (Figure 1 left). This creates a “capability-integration paradox”: LLMs excel at feature exploration but struggle to integrate seamlessly with parameter tuning. To combine LLM-based FE with BO-based HPO without reverting to greedy decoupling, we advocate an interleaved optimization, which raises two further challenges: C2: Collaborative FE–HPO optimization. How can we jointly optimize LLM-based FE and HPO in a coupled manner, rather than optimizing each in isolation? and C3: Task-adaptive scheduling. How should we allocate budget between FE and HPO, given that their marginal utility varies across tasks? To address these challenges, we propose CoFEH (Figure 1 right), a novel framework for holistic ML pipeline optimization. CoFEH is designed to unleash the full potential of LLMs in crafting free-form, human-expert-like FE pipelines, while coupling this with a state-of-the-art BO–based HPO. Our contributions are as follows: • For C1 , we propose an LLM-driven tree-of-thought FE pipeline optimizer to explore unconstrained pipeline topologies and operations, achieving truly free-form feature engineering. • For C2 , we introduce a mutual conditioning mechanism, which establishes a bidirectional information flow between the LLM and BO, enabling them to make decisions conditioned on each other to avoid isolated, sub-optimal tuning. • For C3 , we introduce a PUCB-based dynamic optimizer selector that adaptively allocates the optimization budget between FE and HPO, facilitating efficient interleaved optimization. • Experiments on 28 public datasets show that CoFEH outperforms state-of-the-art traditional and LLM-based baselines in both standalone FE and end-to-end pipeline optimization. 2. Background and Motivation 2.1. The Formal Machine Learning Pipeline The standard supervised machine learning pipeline can be formalized as a hierarchical optimization problem. Given a dataset \mathcal{D}=\{(\mathbf{x}_{i},y_{i})\}_{i=1}^{N} where \mathbf{x}_{i}\in\mathcal{X} represents the raw input space and y_{i}\in\mathcal{Y} the target, the objective is to learn a mapping from raw data to target that minimizes the generalization error. This process is typically decomposed into two interconnected sub-problems: Feature Engineering (FE) ( 26 ; 27 ; 28 ; 19 ) of data and Hyperparameter Optimization (HPO) ( 4 ; 10 ; 32 ; 20 ) of ML model. To automate pipeline construction, traditional AutoML systems cast the entire process into a black-box optimization framework, and leverage method like BO ( 47 ; 11 ; 31 ; 30 ) , genetic programming ( 41 ) , reinforcement learning ( 9 ) , and ADMM ( 34 ) to navigate the search space. Complementing the traditional approaches, Large Language Models (LLMs) have introduced a knowledge-driven paradigm, acting as expert-like agents that propose task-adaptive feature transformation and model configurations ( 16 ; 51 ; 18 ; 35 ;

Read the complete text