Skip to content
Quantoros

What is LoRA and how does it work?

LLM & AILevel: FoundationPEFT Methods

What is LoRA and how does it work?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

LoRA (Hu et al., 2022) adds small trainable low-rank matrices alongside frozen original weights:

Original: W ∈ R^(d×k) — frozen
LoRA: W' = W + BA  where B ∈ R^(d×r), A ∈ R^(r×k), rank r << d

r=16, d=4096: 16×4096 + 16×4096 = 131K params (vs 16.7M original)

During inference: merge W' = W + BA (no latency overhead). During training: only optimize A, B.

Key hyperparameters:

  • Rank (r): 4–64 typical. Higher = more expressive, more parameters. Start with 16.
  • Alpha (α): scaling factor, often set to 2×rank. Controls magnitude of LoRA updates.
  • Target modules: which weight matrices to apply LoRA to. Standard: Q, K, V, output projections.
  • Dropout: 0.05–0.1 on LoRA layers to prevent overfitting.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions