Skip to content
Quantoros

Your Transformer runs out of memory on long documents. How do you scale it?

LLM & AILevel: AdvancedTroubleshooting Scenarios

Your Transformer runs out of memory on long documents. How do you scale it?

Hints
  • Track Q, K and V shapes and which positions can interact.
  • Check masking, weight normalization and training versus autoregressive inference.

Answer

This answer is part of mentoring

Advanced questions come with a full breakdown and a mock interview with a working quant researcher. The first call is free.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions