Why can nvidia-smi show GPU memory in use after PyTorch tensors are deleted?
GPU / CUDALevel: IntermediateAllocated versus reserved GPU memory
Why can nvidia-smi show GPU memory in use after PyTorch tensors are deleted?
Hints
- Distinguish live allocations from reusable cached blocks.
- Which memory can the allocator release without losing tensors?
Answer
PyTorch caches freed device allocations for reuse, so process-level memory reported by nvidia-smi includes reserved memory and context overhead, not only live tensors. Compare allocated and reserved counters, inspect remaining references and processes. empty_cache() can release unused cached blocks to other applications, but it does not free live tensors or fix a retained computational graph.
Source and licence: PyTorch contributors — FAQ · BSD-3-Clause · answer: Quantoros editorial