Why do we need to split our data into three parts: train, validation, and test?
Classic MLLevel: FoundationValidation
Why do we need to split our data into three parts: train, validation, and test?
Hints
- Which observations must remain independent during evaluation?
- Compare random, grouped and temporal splits. Where are hyperparameters selected?
Answer
The training set is used to fit the model, i.e. to train the model with the data. The validation set is then used to provide an unbiased evaluation of a model while fine-tuning hyperparameters. This improves the generalization of the model. Finally, a test data set which the model has never "seen" before should be used for the final evaluation of the model. This allows for an unbiased evaluation of the model. The evaluation should never be performed on the same data that is used for training. Otherwise the model performance would not be representative.
Source and licence: Alexey Grigorev & contributors — Data Science Interviews · CC-BY-4.0