Why can different PyTorch data-loader workers produce identical random augmentations?
GPU / CUDALevel: IntermediateData-loader worker randomness
Why can different PyTorch data-loader workers produce identical random augmentations?
Hints
- Which random-number libraries does your dataset call?
- A reproducible run still needs distinct worker streams.
Answer
Workers may inherit or reuse random state from NumPy or another library, especially with fork-based startup. PyTorch's worker seeding does not automatically configure every external generator. Initialize those generators per worker from the worker seed, and account for persistent workers and epochs. Test both reproducibility between runs and diversity between workers.
Source and licence: PyTorch contributors — FAQ · BSD-3-Clause · answer: Quantoros editorial