Tail-aware N-version Machine Learning Models for Reliable API Recommendation Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom

Aoi Matsuda, Fumio Machida, David Lo · 2026

Read Tail-aware N-version Machine Learning Models for Reliable API Recommendation Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom by Aoi Matsuda, Fumio Machida, David Lo (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27647.txt.

Introduction

Matsuda, Machida, and Lo propose NvRec, a tail-aware API recommendation method that runs several model versions and filters unreliable outputs. Code data sets often over-represent common APIs and under-represent rare ones, so tail APIs get unreliable recommendations. NvRec profiles each model's performance on individual APIs with their tail properties, then filters unreliable recommendations at inference time. The best three-model mix reaches 83.8 percent accepted with about 81 percent rejected, while the five-model mix reaches about 83 percent accepted with 69 percent rejected, a better balance.

Catalog record

CREATORAoi Matsuda, Fumio Machida, David Lo
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55024
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27647

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Tail-aware N-version Machine Learning Models for Reliable API Recommendation Conference: The 30th International Conference on Evaluation and Assessment in Software Engineering; 9–12 June, 2026; Glasgow, Scotland, United Kingdom Aoi Matsuda, Fumio Machida, David Lo Abstract . Machine learning (ML)-based API recommendation helps developers efficiently identify suitable APIs to complement the application code. However, code datasets used to train ML models often exhibit a long-tail distribution, leading to unreliable API recommendations, especially for infrequently used API methods at the tail of the distribution. To address this issue, we propose N-version API Recommendation (NvRec), which leverages N different versions of ML models to enhance the reliability of API sequence recommendations by suppressing unreliable outputs entailing tail APIs. NvRec leverages a set of available ML models and profiles their performance on individual API methods with their tail properties. The generated model profile is used at inference time to filter out unreliable API recommendations and determine the final output. We implement NvRec using five API recommendation models, including CodeBERT, CodeT5, MulaRec, UniXcoder, and CodeT5+, and evaluate it on a public benchmark dataset constructed from compilable Java projects. For the three-version NvRec, we find that the combination of CodeT5, MulaRec, and UniXcoder achieves the highest true accept rate of 83.8%, with a rejection rate of 80.7%, when majority voting is restricted to highly reliable candidates. In contrast, the five-version configuration achieves its highest true accept rate of 83.1% with simple majority voting, while reducing the rejection rate to 69.0%. Overall, the five-version configuration offers a better balance between true accept rate and rejection rate. 1. Introduction Application Programming Interfaces (APIs) define the methods for accessing the functionality provided by software libraries and frameworks. Software developers call library functions through the API to efficiently implement the application code. However, to identify appropriate methods in a given application context, developers should comprehend the functionality of individual APIs from their documentation ( 14 ; 3 ) , which is prohibitively demanding when there are a vast number of available APIs. Although many coding tasks can be automated by AI and coding agents ( 9 ; 10 ) , human developers still need to be responsible for the code to ensure reliability. To support developers in finding appropriate API methods, several recommendation techniques have been presented. The conventional API recommendation techniques are largely divided into single-API recommendations ( 8 ) and API method sequence recommendations ( 5 ; 11 ) . For example, Heinemann et al. ( 8 ) proposed a single API recommendation technique that analyzes identifiers, such as variable names and method names, in the source code and recommends a relevant API method based on the code’s context. For the API method sequence recommendation, Gu et al. ( 5 ) proposed DeepAPI, which uses a deep learning model to generate API method sequences from natural language queries. Recently, Large Language Models (LLMs) have also been used for API recommendation ( 11 ; 19 ; 20 ; 24 ) . For instance, Irsan et al. ( 11 ) achieved higher-precision API method sequence recommendation by inputting both natural language queries and source code into an ML model. Ma et al. presented CAPIR ( 15 ) , which decomposes coarse-grained requirements into fine-grained ones and composes API method sequences by retrieving relevant APIs with an LLM. Although machine learning (ML)-based API recommendations achieve remarkable performance, ML models inherently depend on large source code datasets for training. The distribution of datasets for programming tasks often exhibits a long-tail property, which poses a challenge to the quality of API recommendations, especially for tail API methods ( 27 ) . Recommendation results containing tail API methods tend to be unreliable. Adopting such unreliable API recommendations can introduce bugs into the resulting software, leading to additional development and debugging costs. To address the gap, this paper proposes the N-version API method sequence Recommendation (NvRec) to provide reliable API method sequence recommendations by leveraging the diversity of multi-version ML models to filter out unreliable outputs that contain tail API methods. Inspired by N-version programming ( 1 ) and the N-version ML system architecture ( 16 ) , NvRec uses multiple versions of ML models to produce reliable outputs by aggregating their individual inference results. Since each ML model has a different predictive capability, we prepare the model profile , which records the performance of individual ML models and the tail property for each API method. At inference time, we introduce Tail Analyzer to check if the inputs are categorized as a tail case where an API recommendation is likely to have tail API methods. If the input is classified as a tail case, the request is dropped to suppress unreliable recommendations. If the request passes Tail Analyzer , API method sequence candidates are generated by N-version ML models (multi-version inference). Next, unreliable candidates are excluded using the model profile (filtering). Finally, the recommendation result is selected from the remaining candidates by a specific output decision method (output decision). By leveraging the diversity of ML models and filtering out unreliable API recommendations with tail APIs, NvRec can recommend only reliable API method sequences. We implemented NvRec using several ML models for API recommendation tasks. CodeBERT ( 4 ) , CodeT5 ( 26 ) , and MulaRec ( 11 ) are chosen from the previous studies, while UniXcoder ( 7 ) and CodeT5+ ( 25 ) are fine-tuned for the same API recommendation task. The reliability of the API recommendation sequence is evaluated by the accuracy on accepted outputs (true accept rate), the recommendation rejection rate, and the false rejection rate, using the dataset released by Zhou et al. ( 27 ) , constructed from the 50K-C dataset of compilable Java projects. The evaluation results showed that NvRec achieved a remarkable accuracy of 83.8% by dropping 80.7% of API recommendation candidates, surpassing the best single model, MulaRec (46.3%). The contributions of this paper are as follows: • We propose N-version API Recommendation (NvRec) , a framework for generating reliable API sequence recommendations by filtering tail input cases and aggregating multiple inference results from diverse ML models. • Through the empirical study, we demonstrate the effectiveness of TailAnalyzer, which improves the reliability of individual API recommendation models by filtering the tail cases. • We show that the three-version and five-version NvRec achieve significantly higher reliability than any single model on the extended 50K-C dataset. In the best configuration, the three-version NvRec with CodeT5, MulaRec, and UniXcoder achieved 83.8% true accept rate , with a 80.7% rejection rate and a 32.2% false rejection rate . • By comparing different filtering and output decision methods, we derive insights into selecting appropriate methods to achieve a higher true accept rate while reducing rejection rate and false rejection rate. The remainder of this paper is organized as follows. Section II provides background on the API method sequence recommendation. Section III discusses related work. Section IV presents a performance comparison of ML Models for API method sequence recommendation. Section V details our proposed method. Section VI presents the evaluation results. Section VII discusses threats to validity. Finally, Section VIII concludes the paper and outlines future work. 2. Background 2.1. API Usage and Challenges An API provides an interface for using the functionality

Read the complete text