How does a dense layer with m inputs and n outputs affect a training memory budget?
GPU / CUDALevel: FoundationDense-layer memory budgeting
How does a dense layer with m inputs and n outputs affect a training memory budget?
Hints
- Start with the matrix shape, not the number of layers.
- Which additional tensors appear only during training?
Answer
Its weight matrix contains m times n elements, plus an optional n-element bias. Multiply by storage precision, then include gradients, optimizer state, activations and possible master weights. The weight count is quadratic only when both widths scale together. A model that fits for inference may therefore fail during training, and mixed precision does not make every state use the same datatype.
Source and licence: PyTorch contributors — FAQ · BSD-3-Clause · answer: Quantoros editorial