OpenAutoNLU: Open Source AutoML Library for NLU

Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin · 2026

Read OpenAutoNLU: Open Source AutoML Library for NLU by Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-01824.txt.

Introduction

Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, and colleagues package automated machine learning for language understanding. Their 2026 library, OpenAutoNLU, covers text classification and named entity recognition, two core language tasks. Picking the right training recipe for each data set usually takes expert time and manual setup.

The library chooses the training regime from the data itself, with no manual configuration from the user. Built-in diagnostics flag data quality issues before training begins in earnest. Configurable checks also catch inputs that drift far from the training distribution. Language model features plug into the same low-code interface, which keeps the workflow short. The code and a demo app are public, so teams can test the library on their own text tasks with minimal code.

Catalog record

CREATORGrigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29512
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.01824

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

OpenAutoNLU: Open Source AutoML Library for NLU Grigory Arshinov, Aleksandr Boriskin, Sergey Senichev, Ayaz Zaripov, Daria Galimzianova, Daniil Karpov, Leonid Sanochkin Abstract OpenAutoNLU is an open-source automated machine learning library for natural language understanding (NLU) tasks, covering both text classification and named entity recognition (NER). Unlike existing solutions, we introduce data-aware training regime selection that requires no manual configuration from the user. The library also provides integrated data quality diagnostics, configurable out-of-distribution (OOD) detection, and large language model (LLM) features, all within a minimal low-code API. OpenAutoNLU source code is available here 1 1 1 https://github.com/mts-ai/OpenAutoNLU , the demo app is accessible here https://openautonlu.dev . † † * Equal contribution. † † \dagger Work done while at MWS AI 1 Introduction Text classification and NER are foundational tasks in natural language processing (NLP), underpinning applications from intent detection and sentiment analysis to information extraction and document categorization. Despite their ubiquity, deploying effective models for these tasks remains challenging: practitioners must navigate a complex landscape of competing approaches—full fine-tuning of pretrained transformers, few-shot learning methods, classical machine learning with embeddings—each suited to different data regimes and resource constraints. Further complications arise from data quality issues, hyperparameter sensitivity, and the need for OOD detection in production settings. Automated machine learning (AutoML) offers a promising solution by automating model selection, hyperparameter tuning, and pipeline design, yet existing frameworks exhibit significant gaps when applied to NLP workloads. Figure 1: Text Classification Training Pipeline and Inference Pipeline flow We identify two critical gaps in the current AutoML landscape for NLP. First , ease of use: many frameworks require non-trivial configuration, expose complex abstractions (e.g., table-based predictors or modality-specific modules), and lack simple, unified interfaces for common NLP tasks. Second , NLP-centric design: existing systems do not natively integrate (a) automatic selection of training regimes (full fine-tuning vs. few-shot methods) based on dataset size and label distribution, (b) text-specific data-quality assessment, and (c) unified support for both text classification and NER within a single coherent API. In this work, we present OpenAutoNLU, an open-source AutoML library designed specifically for NLP. OpenAutoNLU addresses the identified gaps through a text-first architecture that emphasizes simplicity, automation, and reproducibility, and demonstrates competitive or superior performance compared to existing AutoML frameworks in standard intent classification benchmarks. The library provides the following key contributions : Automatic training-regime selection, Integrated data-quality tools, Configurable OOD detection, Unified API for classification and NER, Minimal configuration. 2 Background H2O Light AutoML Auto Gluon Auto Intent OpenAuto NLU Approach TAML & Word2Vec TAML & embeddings Encoder finetuning Embeddings Encoder finetuning Scarce train data \times Has small-data modes Adaptable Adapted for small datasets Designed for classes with \geq 2 examples Custom search pipeline Flexible API HP presets HP presets Presets & customizable configs Universal customizable preset LLM Integrations \times \times \times Zero-shot classification Train data augmentations and test set generation OOD detection \times \times \times Train examples required, one method Train examples are optional, configurable Optimized for NLP production Requires preprocessing; Java, Python, R support \times ✓ \times ONNX native; inference in two lines Data quality evaluation \times \times \times \times ✓ Named Entity Recognition \times \times ✓ \times ✓ Embedding prompting \times \times \times ✓ ✓ Table 1: Comparison of AutoML frameworks for NLP tasks. Figure 2: This graph illustrates the ratio between performance in macro F1 on classification tasks and time in seconds took to train the solution. Measures are averaged between four different text classification datasets. Recently, large language models (LLMs) have also been proposed as general-purpose solutions for text classification and intent understanding, often via prompting or lightweight adaptation. While such models can provide strong zero- and few-shot performance 21 , they typically incur substantial computational and monetary costs, as well as higher inference latency due to their size and deployment requirements (e.g., multiple GPU-backed serving or external API calls). These characteristics make LLM-based approaches less suitable for many practical scenarios where models must be deployed at scale 18 , integrated on-premise, or executed under strict latency and resource constraints. In contrast, the AutoML frameworks considered in this work aim to deliver competitive performance using comparatively lightweight architectures and training pipelines, which are more amenable to efficient, low-latency deployment. To our knowledge, some solutions for AutoML have previously been introduced. For example, AutoIntent 2 is an AutoML framework specifically designed for intent classification. It follows an embedding-centric design, where pretrained sentence encoders are combined with either classical classifiers or neural models, followed by automatic decision threshold optimization. AutoIntent supports supervised out-of-distribution (OOD) detection, multi-label classification, and multiple training presets that trade off quality and computational cost. In our experiments, we evaluated classic-light (default), classic-medium, nn-medium, nn-heavy and transformers-heavy AutoIntent presets. AutoGluon 7 is a general-purpose AutoML framework originally developed for tabular data, relying heavily on multi-layer ensembling, stacking, and bagging of heterogeneous models (tree-based models, linear models, and neural networks). LightAutoML 19 and H2O AutoML 11 are also mainly designed for tabular data. Text inputs are handled by simple Word2Vec-based vectorization and subsequently processed by standard tabular models. All experiments with these frameworks were used with default configurations. Feature overview for all frameworks can be found in the Table 1 . 3 OpenAutoNLU OpenAutoNLU is a library in the Python language that is designed to be used in low-code environments. Its simple API allows for quick NLU-model prototyping as well as for robust and flexible production model training. Architecturally, it is a collection of classes that are chained by auto pipelines. The diagram can be found in the Figure 1 . Novelty The principal novelty of OpenAutoNLU lies in its resolution of the data-aware method : rather than exposing the user to a menu of algorithms and hyperparameter grids, the library inspects the label distribution of the supplied dataset and deterministically selects the training method that best fits the data regime. Three regimes are distinguished by the minimum per-class sample count n_{\min} (which were computed on our internal benchmarks): (i) 2~\leq~n_{\min}~\leq~5 — AncSetFit 14 , an anchor-based few-shot method that leverages human-readable class descriptions and triplet-loss contrastive learning; (ii) 5~<~n_{\min}~\leq~80 — SetFit 16 , a sentence-transformer–based few-shot learner with a logistic-regression head; (iii) n_{\min}~>~80 — full transformer fine-tuning 23 with Optuna-driven 1 hyperparameter optimisation. This design means that a practitioner can move from two labelled examples per intent to a production-grade classifier without changing a single line of client code. A second distinctive feature is the integrated out-of-distribution (OOD) detection layer. Each training method has a companion OOD variant—Marginal Mahalanobis distance for the fine-tuning regime, Maximum Softmax Probability for SetFit, and a logit-based “outOfScope” class option—all selectable through a single ood_method configuration flag. Third, OpenAutoNLU ships with an LLM-powered data augmentation and synthetic test generation subsystem: when the number of labelled examples is low, the pipeline can call an external language model to synthesise additional training or evaluation samples, guided by automatic domain analysis. 3.1 Structure Figure 1 shows the high-level module layout. At the top level, the auto_classes module exposes the four public pipeline classes. Each pipeline inherits from AbstractTrainingPipeline or AbstractInferencePipeline , which manage the lifecycle stages: data loading, data processing, (optional) data quality evaluation, method resolution, training, evaluation, and model export. The methods module contains the training algorithms, each subclassing a

Read the complete text