What is LoRA and how does it work?
LLM & AILevel: FoundationPEFT Methods
What is LoRA and how does it work?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
LoRA (Hu et al., 2022) adds small trainable low-rank matrices alongside frozen original weights:
Original: W ∈ R^(d×k) — frozen
LoRA: W' = W + BA where B ∈ R^(d×r), A ∈ R^(r×k), rank r << d
r=16, d=4096: 16×4096 + 16×4096 = 131K params (vs 16.7M original)
During inference: merge W' = W + BA (no latency overhead). During training: only optimize A, B.
Key hyperparameters:
- Rank (r): 4–64 typical. Higher = more expressive, more parameters. Start with 16.
- Alpha (α): scaling factor, often set to 2×rank. Controls magnitude of LoRA updates.
- Target modules: which weight matrices to apply LoRA to. Standard: Q, K, V, output projections.
- Dropout: 0.05–0.1 on LoRA layers to prevent overfitting.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0