Skip to content
Quantoros

Why is an estimator with partialfit insufficient by itself to make a training system out-of-core?

Classic MLLevel: IntermediateComplete streaming workflow

Why is an estimator with partial_fit insufficient by itself to make a training system out-of-core?

Hints
  • Follow memory use before the estimator.
  • Streaming is an end-to-end property.

Answer

Data reading and feature extraction must also operate within bounded memory. A loader that materializes the entire dataset defeats incremental fitting even if the final estimator supports batches.

Source and licence: scikit-learn 1.7.2 — scaling strategies · BSD-3-Clause · answer: Quantoros editorial

Related questions