Fortress: A Case Study in Stabilizing Search Recommendations via Temporal Data Augmentation and Feature Pruning

1 st Milind Pandurang Jagre, 2 nd Jia Huang, 3 rd Dayvid V. R. Oliveira, 4 th Zhinan Cheng, 5 th Babak Seyed Aghazadeh, 6 th Puja Das, 7 th Chris Alvino, 8 th Jinda Han, 9 th Kailash Thiyagarajan · 2026

Read Fortress: A Case Study in Stabilizing Search Recommendations via Temporal Data Augmentation and Feature Pruning by 1 st Milind Pandurang Jagre, 2 nd Jia Huang, 3 rd Dayvid V. R. Oliveira, 4 th Zhinan Cheng, 5 th Babak Seyed Aghazadeh, 6 th Puja Das, 7 th Chris Alvino, 8 th Jinda Han, 9 th Kailash Thiyagarajan (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-15299.txt.

Introduction

Milind Pandurang Jagre and eight others present Fortress, a framework for steadying search recommendations against temporal drift. Signals that shift over time make model scores jumpy, harming multi-stage systems that need steady predictions. Fortress collects historical snapshots, flags samples with unstable scores, spots and prunes wobble-causing features, and retrains on stable signals only. The method tames volatile engagement signals while keeping their forecast value, yielding models that are both more stable and more true in testing.

Catalog record

CREATOR1 st Milind Pandurang Jagre, 2 nd Jia Huang, 3 rd Dayvid V. R. Oliveira, 4 th Zhinan Cheng, 5 th Babak Seyed Aghazadeh, 6 th Puja Das, 7 th Chris Alvino, 8 th Jinda Han, 9 th Kailash Thiyagarajan
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE12103
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.15299

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Fortress: A Case Study in Stabilizing Search Recommendations via Temporal Data Augmentation and Feature Pruning 1 st Milind Pandurang Jagre, 2 nd Jia Huang, 3 rd Dayvid V. R. Oliveira, 4 th Zhinan Cheng, 5 th Babak Seyed Aghazadeh, 6 th Puja Das, 7 th Chris Alvino, 8 th Jinda Han, 9 th Kailash Thiyagarajan Abstract In search and recommendation systems, predictive models often suffer from temporal instability when certain input features introduce volatility in output scores. This instability can degrade model reliability and user experience especially in multi-stage systems where consistent predictions are critical for downstream decision making. We introduce Fortress, a general framework for enhancing model stability and accuracy by identifying and pruning features that contribute to inconsistent prediction scores over time. Fortress leverages historical snapshots temporally partitioned datasets capturing score fluctuations for the same entity across periods and follows a four-step process: (1) collect historical snapshots, (2) identify samples with unstable predictions, (3) isolate and remove instability- inducing features, and (4) retrain models using only stable features. While semantic features from LLMs and BERT-based models improve generalization, they often lack full query or entity coverage. Engagement-based features offer strong predictive power but tend to introduce temporal instability. Fortress mitigates this trade-off by suppressing the volatility of engagement signals while retaining their predictive value leading to more stable and accurate models. We validate Fortress on a query-to-app relevance model in a large-scale app marketplace. Offline experiments demonstrate notable improvements in prediction stability (measured by Coefficient of Variation) and classification performance (measured by PR-AUC). I Introduction Recommendation systems must provide not only relevant results, but also consistent and predictable ones. Instability in model predictions where the same query-app pair yields significantly different scores across time can lead to poor user experience and undermine trust in the system. While recent advancements in LLM-based features have improved contextual relevance [2, 4, 10], engineered features derived from organic user engagement behavior (clicks, searches, etc.) continue to play a crucial role [7, 8, 12, 14, 15]. However, these features can introduce temporal inconsistencies that affect score stability. Model instability often arises when certain features cause the output score to vary across temporal snapshots, leading to inconsistent user experiences. In search-based recommendation systems, these score fluctuations can cause items to alternate between being shown and not shown in search results. Such instability can affect the overall quality of the recommendations, leading to a less consistent and predictable user experience. In this work, we present Fortress, a generalizable framework for improving model prediction stability through feature selection by leveraging historical snapshots of model inputs and outputs over time. The core insight is that some features though potentially predictive in isolated snapshots may introduce volatility across time. By identifying and pruning such features, we can enhance both model robustness and accuracy. This method fits into the broader class of “wrapper” feature selection methods, where a feature subset is selected based on validation performance, as introduced in [6]. A review of feature selection methods can be found in [11]. We evaluate this method on a relevance prediction task for an app marketplace, where the goal is to classify query-app pairs as relevant or not. While this case study grounds the methodology, the technique is broadly applicable to any scenario where a scoring model evaluates the same entity over time, such as user-item recommendations. Our results show improved prediction consistency as measured by decreased Coefficient of Variation (CV) [5] and improved classification performance as measured by increased PR-AUC. II THE FORTRESS FRAMEWORK Our method assumes that certain features may lead to unstable predictions when used in model scoring across time. We define a feature as unstable if its inclusion causes prediction volatility across historical snapshots for the same entity. Historical snapshots are temporally partitioned datasets contiguous intervals over a time period allowing us to evaluate prediction consistency for entities scored multiple times. Fig. 1: Representation of multi-snapshot approach with data sampled across different historical points, and disjoint samples between training, validation, and test sets. We demonstrate the method using a dataset of query-app pairs from a major commercial app marketplace. Human raters label each pair as BAD, ACCEPTABLE, GOOD, or EXCELLENT. To simplify evaluation, we treat this as a classification task where the model learns to filter out BAD pairs. We utilized a quarter of the data, consisting of daily snapshots, for training and validation purposes. The entire dataset was divided into training, test, and validation datasets, following the industry standard of 70%, 15%, and 15% respectively. The entire dataset consisted of approximately 40% pairs labeled as BAD, while the remaining 60% were divided equally among pairs labeled as ACCEPTABLE, GOOD, or EXCELLENT. Our evaluation metrics include PR-AUC to assess classification performance, and Coefficient of Variation (CV) to measure score stability, defined as standard deviation divided by mean for prediction scores across snapshots. The proposed method proceeds as follows: 1. Obtain multiple snapshots over time. Each entity (e.g., query-app pair) and its snapshot data is assigned to a single data partition. 2. Identify unstable samples with high score variance across snapshots. 3. Identify and remove features that contribute to instability while offering limited performance gain. 4. Retrain the model using the reduced feature set and all snapshots. To determine which features to remove, we first identify samples corresponding to the 75th percentile highest CV across temporal snapshots. For these high-variance samples, we compute the CV for each feature individually. Features with the highest CVs are considered candidates for pruning. However, we only proceed with feature removal if doing so results in statistically significant improved PR-AUC on the validation set as compared to baseline. We employ a greedy search approach to compute the CV by iteratively removing features deemed for pruning during model retraining without those features. This process continues until the pruned feature list is exhausted, and we achieve a statistically significant PR-AUC lift. This is particularly important in systems with multi-stage retrieval and ranking pipelines, where instability in prediction scores can propagate through downstream stages and cause significant shifts in overall system behavior. III EXPERIMENTS AND RESULTS We evaluated our method using an XGBoost classifier [1] to predict an app relevance in the context of a query, by training on largely, two categories of features, (a) Semantic Representation (SR): LLM-based and DistilBERT based features and (b) Engagement features. We used two types of SR features. The first is an LLM-based [13] feature designed to capture query-to-app similarity independent of engagement signals. For this, the system employs a proprietary Foundation Model [3], which leverages query and app metadata to generate relevance scores. The second is based on a DistilBERT model [9] in a two-tower architecture trained with triplet loss, where the model learns to embed queries and items such that relevant pairs are closer and irrelevant pairs are pushed further apart in the embedding space. The engagement features are non-personalized and based on highly aggregated historical user interactions with the query-app pair (clicks, searches etc.). Our model consisted of around 25 features, which were refreshed several times daily and are the main source of predicted score instability. XGBoost was selected for its speed and robustness to noise. As is typical in classification modeling, we interpret the output score

Read the complete text