How do you serve multiple LoRA adapters efficiently from one base model?
LLM & AILevel: IntermediateAdvanced Topics
How do you serve multiple LoRA adapters efficiently from one base model?
Hints
- Which parameters change and which stay frozen?
- Separate memory for weights, gradients, optimizer states and activations.
Answer
Running separate inference servers for each LoRA adapter is wasteful — 50 task-specific adapters would require 50× the GPU memory. Efficient multi-adapter serving:
Option 1 — Dynamic adapter loading: load base model once; swap LoRA adapter weights per request. Simple but adds latency for swapping (~10–50ms). Works when request rate per adapter is low.
Option 2 — S-LoRA / LoRA-as-a-Service: batch requests from multiple adapters together. During a forward pass, apply different LoRA weights to different sequences in the same batch:
- Unified paging: store all LoRA adapter weights in a pool (like KV cache paging)
- Scheduled merging: fuse applicable adapters into base weights for the current batch
- Implementations: S-LoRA (vLLM community fork), Punica
Option 3 — Merged inference for top adapters: for your highest-traffic adapters, pre-merge the adapter into base weights (W_merged = W_base + BA) and serve as a standalone model. No runtime overhead.
Decision rule:
- < 10 adapters, moderate traffic: dynamic loading
- 10–100 adapters: S-LoRA or request batching by adapter
-
100 adapters: multi-tier (merged for top 5, dynamic for rest)
Source and licence: JustInCache & contributors — Awesome AI Interviews · Apache-2.0