Why may LightGBM feature parallelism fail to solve a dataset-too-large-for-one-machine problem?
Classic MLLevel: IntermediateLightGBM · Features
Why may LightGBM feature parallelism fail to solve a dataset-too-large-for-one-machine problem?
Hints
- Identify what is replicated and what is distributed.
- Splitting feature work need not split row storage.
Answer
Its documented feature-parallel design keeps the full data on every worker while sharing split search. It reduces some compute/communication costs, not per-worker row storage; consider data parallelism for that bottleneck.
Source and licence: LightGBM · Features · MIT · answer: Quantoros editorial