Skip to content
Quantoros

How does a dense layer with m inputs and n outputs affect a training memory budget?

GPU / CUDALevel: FoundationDense-layer memory budgeting

How does a dense layer with m inputs and n outputs affect a training memory budget?

Hints
  • Start with the matrix shape, not the number of layers.
  • Which additional tensors appear only during training?

Answer

Its weight matrix contains m times n elements, plus an optional n-element bias. Multiply by storage precision, then include gradients, optimizer state, activations and possible master weights. The weight count is quadratic only when both widths scale together. A model that fits for inference may therefore fail during training, and mixed precision does not make every state use the same datatype.

Source and licence: PyTorch contributors — FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions