Skip to content
Quantoros

How should several independent model checkpoints be exposed behind one vLLM-compatible API endpoint?

LLM & AILevel: IntermediatevLLM model routing

How should several independent model checkpoints be exposed behind one vLLM-compatible API endpoint?

Hints
  • Separate independent checkpoints from adapters on one base.
  • Where should model-name routing and resource limits live?

Answer

In the referenced FAQ, a server instance serves one base model; deploy instances and route requests through a gateway that maps model names to backends. Multi-LoRA adapters of one base model are a different feature, not arbitrary multi-checkpoint hosting. Check the installed version's capabilities, isolate resource budgets and validate the selected model rather than silently routing to a default.

Source and licence: vLLM contributors — FAQ · Apache-2.0 · answer: Quantoros editorial

Related questions