On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning.

Xu Y, Goodacre R. · 2018

Read On Splitting Training and Validation Set: A Comparative Study of Cross-Validation, Bootstrap and Systematic Sampling for Estimating the Generalization Performance of Supervised Learning. by Xu Y, Goodacre R. (2018) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc6373628.txt.

Description

Xu and Goodacre present a comparative study in the Journal of Analysis and Testing. They test how the split of data into training and test parts changes judged model quality. Good checks matter, since scores on training data alone can hide overfitting and flatter a weak model.

They made nine simulated datasets and built two kinds of classifiers on each split. The splits used cross-validation, bootstrap forms, and fixed sampling plans such as Kennard-Stone and SPXY. On small datasets, judged quality missed true test quality by a wide gap for all methods. The gap shrank as datasets grew and models neared stable behavior. Fixed sampling plans gave the worst estimates. The split sizes also needed a fair balance to judge quality well.

Catalog record

CREATORXu Y, Goodacre R.
YEAR2018
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEEurope PMC

Source: https://europepmc.org/articles/PMC6373628

Browse the full library · About the Tower of Babel