Interval Regression: A Comparative Study with Proposed Models Tung L Nguyen, Toby Dylan Hocking Abstract Regression models are essential for a wide range of real-world applications. However, in practice, target values are not always precisely known; instead, they may be represented as intervals of acceptable values. This challenge has led to the development of Interval Regression models. In this study, we provide a comprehensive review of existing Interval Regression models and introduce alternative models for comparative analysis. Experiments are conducted on both real-world and synthetic datasets to offer a broad perspective on model performance. The results demonstrate that no single model is universally optimal, highlighting the importance of selecting the most suitable model for each specific scenario. keywords interval regression, survival analysis, supervised machine learning. 1 Introduction Need for Interval Regression over Standard Regression Regression models are widely used in real-world applications. However, due to some reasons, such as errors in measurement, the data collection procedures, or human factors, the target value in many scenarios is not a single value but rather an interval of acceptable values. For instance, in medical research ( Hawkins 2004 ; Fazil Baksh et al. 2006 ) , the precise survival time of a patient is often uncertain, but it is known that the patient survives for at least a certain number of years. In economic forecasting ( Abbara and Zevallos 2018 ) , and environmental studies ( Ellison et al. 2014 ) , precise measurements at a specific time period are unreliable, leading researchers to use the mean and confidence interval of measurements over time. Similarly, in engineering ( Levitin et al. 2011 ) , multiple measurements of a single statistic are taken, and the mean along with the confidence interval is considered as the target. These uncertainties motivate the study of Interval Regression. The difference between Interval Regression and Standard Regression In the Standard Regression setting, each instance is associated with a single target value. In Interval Regression setting, each instance is associated with an interval of acceptable values, called target interval . There are four types of target intervals (y_{l},y_{u}) : uncensored ( -\infty0 is the margin length ( \epsilon=1 by default) and the loss type p=2 are chosen. Cheng et al. 2023 proposed a model called Regression with Interval Targets (RIT), which is essentially the same as the above model, but uses a loss type of p=1 instead. Tree The Tree model, called Maximum Margin Interval Trees (MMIT), was introduced by Drouin et al. 2017 as a nonlinear model to Interval Regression. The prediction is given by: \hat{y}=\mathcal{T}(\mathbf{x}) where \mathcal{T} represents the tree model. The tree architecture follows the same structure as the regression decision tree CART (Classification And Regression Tree) ( Breiman et al. 1984 ) . The only difference lies in the regression value for each leaf: instead of taking the mean of all targets like CART does, it chooses a constant value that minimizes the mean Hinge Error ( 1 ) between this constant and the target intervals. They also introduce an alternative Interval Regression loss function, called Hinge Linear Error , which is defined by Equation ( 1 ) with p=1 . AFT Model in XGBoost The AFT model in XGBoost, introduced by Barnwal et al. 2022 , is an ensemble tree-based model designed for survival analysis. The model makes predictions as follows: \hat{y}=\mathbf{T}(\mathbf{x}), where \mathbf{T} represents the ensemble of trees in XGBoost. Although it belongs to the AFT model family, this model can handle all non-negative target intervals. To extend its applicability to real-valued target intervals (y_{l},y_{u})\in\mathbf{R}^{2} , Barnwal et al. 2022 apply an exponential transformation to the original interval: (y_{l},y_{u})\longrightarrow(\exp y_{l},\exp y_{u}). After making a prediction y_{p} in the exponential space, the result is mapped back to the original scale using the logarithm transformation: \hat{y}=\log y_{p} 2.2 Proposed Models K-Nearest Neighbors KNN is a classical model for Standard Regression, which motivates its consideration in the Interval Regression setting. The Interval Regression procedure follows the same steps as the standard KNN regression model ( Fix and Hodges 1985 ) , with the key difference being how the regression value is determined from the k nearest neighbors. We propose utilizing the MMIT regression function. Specifically, we treat the set of k nearest neighbors as a single leaf in a MMIT, where the regression value is a constant that minimizes the mean Hinge Error ( 1 ) between this constant and the k target intervals. The prediction \hat{y} based on the target intervals of the k nearest neighbors, denoted as \{(y_{l}^{i},y_{u}^{i})\}_{i=1}^{k} , is formulated as: \hat{y}=\argmin_{c}\frac{1}{k}\sum_{i=1}^{k}l\big(c,(y_{l}^{i},y_{u}^{i})\big) Random Forest The idea of using ensembles of MMITs is introduced in Chapter 6 (Discussion and Conclusions) of ( Drouin et al. 2017 ) . Since MMIT is essentially a decision tree regressor for the Interval Regression setting, its extension to construct a Random Forest for Interval Regression is straightforward. This proposed model is referred to as the Maximum Margin Interval Forest (MMIF), as it is composed of multiple MMITs. The operation of MMIF follows the same principles as the standard Random Forest regressor ( Breiman 2001 ) , with the key distinction being that MMITs are used instead of conventional decision trees. MLP Since MLP is a commonly used neural network and serves as a universal approximator in Standard Regression models ( Hornik et al. 1989 ) , its application in Interval Regression is worth considering. MLP models for Interval Regression are proposed, where the Squared Hinge Error ( 1 ) is used for training. 3 Experiments Experimental Datasets To ensure a fair and comprehensive comparison across different scenarios, we evaluate all the models on a diverse set of datasets. Specifically, we use 36 real-world datasets from the UCI repository ( Lichman 2013 ) , along with 3 simulated datasets, identical to those in ( Drouin et al. 2017 ) . Each simulated dataset consists of 200 instances with 20 features, where 19 are noise and one is the true feature x . The target intervals are defined based on the linear function of x , \sin(x) , or the absolute value of x . All datasets are available at https://github.com/aldro61/mmit-data . Train/Test Setup Each dataset is divided into 5 similar sized folds. Each fold is used as the test set, while the remaining 4 folds are combined to form the train set. Evaluation Metrics For each train/test pair, the Interval Regression model is trained on the train set and used to make predictions on the test set. The test set is denoted as having m instances with corresponding target intervals \{(y_{l}^{i},y_{u}^{i})\}_{i=1}^{m} , and the set of predictions is \{\hat{y}^{i}\}_{i=1}^{m} . The Mean Squared Hinge Error is given by: \frac{1}{m}\sum_{i=1}^{m}\left(\text{ReLU}(y_{l}^{i}-\hat{y}^{i})\right)^{2}+\left(\text{ReLU}(\hat{y}^{i}-y_{u}^{i})\right)^{2} For each dataset, 5 train/test pairs are used, resulting in 5 error values. From these 5 error values, the mean and standard deviation are computed. Two metrics are then considered: performance and consistency . A smaller mean error indicates better performance, while a smaller standard deviation indicates greater consistency. 3.1 Model Configuration Except for the AFT model in XGBoost, the Squared Hinge Error is used as the evaluation function for both the cross-validation process and model training. All models are implemented using Python, except for the Linear model. The configuration for each model below is applied to a single train/test set pair. Constant (Featureless) This model predicts a single value from the train set using only target intervals. The value is a constant that minimizes the Squared Hinge Error ( 1 ) between this constant and the set of target intervals. Linear The R package penaltyLearning ( Hocking 2024 ) is used to implement the Max Margin Interval Regression model with L1 regularization. The regularization parameter starts at 0.001, increasing by a factor of 1.2 until no features remain, with cross-validation ( cv = 5) on the train set to determine the optimal L1 regularization value. MMIT The MMIT model was implemented using the C++/Python bindings package mmit ( Drouin 2017 ) . A cross-validation ( cv = 5) was used to select the optimal hyperparameters, including max_depth {0, 1, 5, 10, 20, \infty } and min_sample {0, 1, 2, 4, 8, 16, 20}. AFT model in XGBoost The model was implemented using the Python package xgboost . Cross-validation ( cv = 5) was performed to select the optimal hyperparameters, following the same grid search described in Barnwal et al. 2022 : • learning_rate : 0.001, 0.01, 0.1, 1.0 • max_depth : 2, 3, 4, 5, 6, 7, 8, 9, 10 • min_child_weight : 0.001, 0.1, 1.0, 10.0, 100.0 • reg_alpha : 0.001, 0.01, 0.1, 1.0, 10.0, 100.0 • reg_lambda : 0.001, 0.01, 0.1, 1.0, 10.0, 100.0 • aft_loss_distribution_scale : 0.5, 0.8, 1.1, 1.4, 1.7, 2.0 KNN The Euclidean distance metric is applied to normalized features (mean 0, standard deviation 1) to determine the nearest neighbors, implemented using the NearestNeighbors module from the sklearn.neighbors package. A cross-validation ( cv = 5) is used to select the optimal value of K . The candidate values for K range from 1 to \lceil\sqrt{n}\rceil , where n is the number of train instances. MMIF MMIF is an ensemble of 100 MMITs. For each MMIT, two-thirds of the train dataset is randomly selected for training, while the remaining one-third serves as the out-of-bag (OOB) set for validation. Additionally, one-third of the original features are randomly chosen for each MMIT. A cross-validation ( cv = 5) is used to select the optimal hyperparameters for each MMIT including max_depth {2, 5, 10, 15, 20, 25} and min_split_samples {2, 5, 10, 20, 50}. Each MMIT has its own OOB error, which is used to determine the aggregation rule. Let \mathcal{T}_{i} be an MMIT with OOB error e_{i} . The weight assigned to \mathcal{T}_{i} is given by w_{i}=\frac{\frac{1}{e_{i}}}{\sum_{j=1}^{100}\frac{1}{e_{j}}} and the final prediction from MMIF is computed as \sum_{i=1}^{100}w_{i}\mathcal{T}_{i}(\mathbf{x}) where \mathbf{x} is the set of features. MLP A cross-validation ( cv = 5) is used to determine the optimal hyperparameters: • num_layer : {1, 2} • hidden_layer_size : {5, 10, 20} • activation : {ReLU, Sigmoid} The model is trained using the Adam optimizer with a fixed learning rate of 0.001. Figure 2: The mean and standard deviation of the log of test squared hinge errors from simulated datasets. The Linear model performs best when the dataset is linear. In nonlinear datasets, Tree-based models achieve the best performance. Figure 3: The mean and standard deviation of test squared hinge errors for datasets with a high number of features (due to significantly higher error, the Constant model is omitted to improve comparative visualization). Tree-based models generally perform exceptionally well due to their inherent feature selection mechanism. On the other hand, while MLP with ReLU activation function is a more generalized Linear model, it fails to outperform the Linear model. One reason for this is that when a dataset contains a majority of noisy features, MLP cannot effectively reduce their impact on predictions in the same way that a Linear model with L1 regularization can. Figure 4: The mean and standard deviation of test squared hinge errors for datasets with a moderate number of features. In these scenarios, MLP generally performs well. Performance Consistency 1st 2nd 3rd 4th 5th 6th 7th 1st 2nd 3rd 4th 5th 6th 7th constant 2 2 3 1 4 7 20 4 0 2 3 3 8 19 knn 4 4 2 6 6 16 1 4 3 3 7 7 12 3 linear 3 11 6 8 5 5 1 7 7 8 7 7 3 0 mmit 2 10 9 3 9 4 2 2 13 4 8 8 2 2 mmif 15 5 8 7 4 0 0 12 9 9 4 3 2 0 aft_xgb 4 4 3 4 7 5 12 4 1 7 2 3 10 12 mlp 9 3 8 10 4 2 3 6 6 6 8 8 2 3 Table 1: The comparison of the performance and consistency of 7 models across 39 datasets. 4 Discussion Dataset Variations To ensure a fair and comprehensive comparison, the quality of the datasets is crucial. First, the quality of features is considered. The Constant model (or Featureless), being the simplest, is expected to perform the worst across all datasets. However, in 19 datasets, it is not ranked the lowest. Notably, in 8 of these datasets, it is ranked 4th place or higher, suggesting that the features in these datasets provide limited predictive value. Next, the relationship between features and targets is examined. Since Linear models assume a linear relationship, while MMIT assumes a highly nonlinear one, dataset linearity is assessed based on performance and consistency from both Linear and MMIT models. If the Linear model outperforms MMIT with greater consistency, the dataset is more likely to exhibit a linear relationship. However, datasets where Linear performs better but with lower consistency than MMIT are not considered. From that criteria, out of 39 datasets, 15 are identified as more linear than nonlinear, including autohorse, autompg, bodyfat, breasttumor, cholesterol, cleveland, elusage, fishcatch, fruity, meta, pbc, simulated.linear, sleep, and vineyard. Conversely, 15 datasets are more nonlinear than linear, such as autoprice, basketball, cloud, cpu, lowbwt, lymphoma.mkatayama, machine.cpu, mbagrade, pollution, pwlinear, pyrim, servo, simulated.abs, simulated.sin, and strike. This statistic demonstrates the diversity and quality of the datasets, ensuring a fair comparison across different scenarios. Comprehensive Comparison Let’s examine the performance and consistency rankings in Table 1 . Starting with performance, it’s clear that MMIF outperforms other models in most cases. Linear, MMIT and MLP are comparable, while Constant, KNN, and the AFT model in XGBoost show the poorest performance. When it comes to consistency, MMIF stands out as the most consistent. This is expected, as MMIF being an ensemble model, relies on a group of weak MMITs to make the final prediction, which helps improve its stability. Again, Constant, KNN, and the AFT model in XGBoost show the lowest consistency. No single model is optimal for all scenarios As shown in Table 1 , there is no universally optimal model for every scenario. However, if complexity is not a primary concern, and the goal is to select the model with the best overall performance and consistency, MMIF is likely the first choice. Alternatively, if model complexity and training time are critical factors, MMIT or Linear are preferable, as they are less complex than MMIF, require significantly fewer training resources, and still maintain reliable performance. MMIT vs MMIF This is a comparison worth considering. MMIF was introduced with the expectation that an ensemble of MMITs would outperform a single MMIT, and the results confirm this hypothesis. Specifically, across 39 datasets, MMIF is found to outperform MMIT in 28 of them, which constitutes the majority. The reason for this is straightforward: MMIF, being an ensemble of MMITs, is able to reduce overfitting by not relying on a single MMIT. Discussion on the AFT Model in XGBoost and potential future work The AFT model in XGBoost is one of the most complex models in this study, but its performance and consistency do not justify the complexity. It has a limitation in that it can only handle non-negative target intervals. To make the model applicable to a general Interval Regression setting, the target intervals must be transformed using exponential function. This introduces a weakness, as the model becomes highly sensitive when the dataset contains many left-censored target intervals. When predictions are made on the exponential scale, if the predicted value is close to zero, the mapping back to the original scale can result in large negative values, as seen in some test cases where predictions like -10^{6} were made. (-\infty,y_{u})\overset{\text{exp}}{\longrightarrow}(0,\exp y_{u})\overset{\text{predict}}{\longrightarrow}y_{p}\approx 0\overset{\text{log}}{\longrightarrow}\hat{y}\approx-\infty\overset{\text{evaluate}}{\longrightarrow}\text{Big test error} The experiments conducted in ( Barnwal et al. 2022 ) indicate that the AFT model in XGBoost performs among the worst across their six simulated datasets when compared to the Linear model and MMIT. A potential direction for future work is to transform left-censored intervals into interval-censored intervals by replacing all lower bounds of -\infty with a finite value. This adjustment aims to reduce the model’s sensitivity to left-censored intervals and mitigate the tendency to produce excessively large negative predictions. Code Available All experiment code is available in this GitHub repository: https://github.com/lamtung16/ML_IntervalRegression for reproducibility and further exploration of the models used in this study. SUPPLEMENTAL MATERIALS The two models with the highest mean test Hinge Squared Error have been omitted for clearer visualization. constant knn linear mmit mmif aft_xgb mlp auto93 7 6 3 5 1 2 4 autohorse 7 4 2 5 1 6 3 autompg 7 6 2 3 4 5 1 autoprice 7 4 6 2 1 5 3 baskball 6 2 5 3 1 7 4 bodyfat 7 6 2 4 3 5 1 breasttumor 6 1 4 5 2 7 3 cholesterol 3 5 4 6 1 7 2 cleveland 6 4 2 5 3 7 1 cloud 7 6 3 2 5 4 1 cpu 7 6 4 3 1 2 5 echomonths 7 4 2 5 3 6 1 elusage 7 2 1 3 4 5 6 fishcatch 7 6 4 5 3 2 1 fruitfly 1 5 2 6 3 7 4 housing 7 3 6 5 4 2 1 lowbwt 6 5 3 2 1 7 4 lymphoma.mkatayama 1 5 6 3 4 7 2 lymphoma.tdh 3 6 2 4 1 7 5 machine.cpu 7 6 5 3 2 1 4 mbagrade 3 6 5 1 2 4 7 meta 5 1 2 6 4 7 3 pbc 7 5 1 4 2 6 3 pharynx 7 6 2 1 3 4 5 pollution 4 6 3 2 1 7 5 pwlinear 7 6 5 2 3 4 1 pyrim 2 1 6 3 4 5 7 sensory 2 6 3 5 1 7 4 servo 7 6 4 2 5 1 3 simulated.abs 5 4 6 2 1 3 7 simulated.linear 7 6 1 3 2 5 4 simulated.sin 5 7 4 2 1 3 6 sleep 7 1 2 5 3 6 4 stock 7 2 5 3 1 6 4 strike 6 2 4 7 5 1 3 triazines 5 6 7 2 1 3 4 veteran 6 3 2 7 4 5 1 vineyard 7 4 3 6 5 1 2 wisconsin 6 5 4 2 1 7 3 Table 2: The performance ranking of 7 models across 39 datasets. constant knn linear mmit mmif aft_xgb mlp auto93 7 6 4 2 1 3 5 autohorse 7 4 2 5 1 6 3 autompg 7 5 1 4 2 6 3 autoprice 7 3 4 2 1 6 5 baskball 5 4 3 2 1 6 7 bodyfat 7 5 1 4 3 6 2 breasttumor 6 1 2 5 3 7 4 cholesterol 5 6 4 7 1 3 2 cleveland 6 5 1 4 3 7 2 cloud 7 6 3 2 5 4 1 cpu 7 4 5 2 1 3 6 echomonths 7 2 5 4 3 6 1 elusage 7 4 1 2 3 5 6 fishcatch 7 6 2 5 4 3 1 fruitfly 3 5 2 7 1 6 4 housing 7 3 4 5 2 6 1 lowbwt 6 4 3 1 2 7 5 lymphoma.mkatayama 1 6 5 2 3 7 4 lymphoma.tdh 3 6 4 2 1 7 5 machine.cpu 7 6 5 2 3 1 4 mbagrade 6 2 5 1 3 4 7 meta 6 1 2 5 4 7 3 pbc 6 2 3 5 4 7 1 pharynx 6 5 1 3 4 7 2 pollution 4 6 3 2 1 7 5 pwlinear 7 6 5 4 2 3 1 pyrim 1 3 6 4 5 2 7 sensory 1 6 3 4 2 7 5 servo 7 5 4 2 6 1 3 simulated.abs 6 7 5 2 1 3 4 simulated.linear 7 6 1 4 2 5 3 simulated.sin 4 7 6 2 1 3 5 sleep 7 4 2 5 1 6 3 stock 7 1 2 5 3 6 4 strike 5 7 6 3 2 1 4 triazines 1 5 3 6 2 7 4 veteran 7 1 4 3 6 5 2 vineyard 7 4 3 6 5 1 2 wisconsin 4 6 1 3 2 7 5 Table 3: The consistency ranking of 7 models across 39 datasets.