Skip to content
Quantoros

How do you merge multiple LoRA adapters?

LLM & AILevel: IntermediatePEFT Methods

How do you merge multiple LoRA adapters?

Hints
  • Which parameters change and which stay frozen?
  • Separate memory for weights, gradients, optimizer states and activations.

Answer

When you have multiple task-specific LoRA adapters, merging options:

  1. Sequential merge: W = W_base + α₁BA₁ + α₂BA₂ — sum of all adapter contributions. Works well when tasks are not conflicting.

  2. Task arithmetic: perform vector arithmetic in weight space — W_merged = W_base + λ(W_task1 - W_base) + μ(W_task2 - W_base). Can combine capabilities or negate unwanted behaviors.

  3. TIES-Merging: resolves conflicts between adapter weights through magnitude pruning and sign agreement before summing.

  4. DARE: randomly drops (with high probability) and rescales adapter delta weights before merging — reduces interference.

Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0

Related questions