Why is an estimator with partialfit insufficient by itself to make a training system out-of-core?
Classic MLLevel: IntermediateComplete streaming workflow
Why is an estimator with partial_fit insufficient by itself to make a training system out-of-core?
Hints
- Follow memory use before the estimator.
- Streaming is an end-to-end property.
Answer
Data reading and feature extraction must also operate within bounded memory. A loader that materializes the entire dataset defeats incremental fitting even if the final estimator supports batches.
Source and licence: scikit-learn 1.7.2 — scaling strategies · BSD-3-Clause · answer: Quantoros editorial