Skip to content
Quantoros

Why may LightGBM feature parallelism fail to solve a dataset-too-large-for-one-machine problem?

Classic MLLevel: IntermediateLightGBM · Features

Why may LightGBM feature parallelism fail to solve a dataset-too-large-for-one-machine problem?

Hints
  • Identify what is replicated and what is distributed.
  • Splitting feature work need not split row storage.

Answer

Its documented feature-parallel design keeps the full data on every worker while sharing split search. It reduces some compute/communication costs, not per-worker row storage; consider data parallelism for that bottleneck.

Source and licence: LightGBM · Features · MIT · answer: Quantoros editorial

Related questions