Skip to content
Quantoros

What GPU memory does activation checkpointing save, and what cost does it introduce?

GPU / CUDALevel: AdvancedActivation recomputation trade-off

What GPU memory does activation checkpointing save, and what cost does it introduce?

Hints
  • Separate saved activations from weights and optimizer states.
  • What work has to be repeated during the backward pass?

Answer

This answer is part of mentoring

Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.

Source and licence: PyTorch contributors — FAQ · BSD-3-Clause · answer: Quantoros editorial

Related questions