How do you merge multiple LoRA adapters?
LLM & AILevel: IntermediatePEFT Methods
How do you merge multiple LoRA adapters?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
When you have multiple task-specific LoRA adapters, merging options:
-
Sequential merge:
W = W_base + α₁BA₁ + α₂BA₂— sum of all adapter contributions. Works well when tasks are not conflicting. -
Task arithmetic: perform vector arithmetic in weight space —
W_merged = W_base + λ(W_task1 - W_base) + μ(W_task2 - W_base). Can combine capabilities or negate unwanted behaviors. -
TIES-Merging: resolves conflicts between adapter weights through magnitude pruning and sign agreement before summing.
-
DARE: randomly drops (with high probability) and rescales adapter delta weights before merging — reduces interference.
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0