Serve an open model with vLLM: a deployment tutorial

vLLM has become the default way to serve open-weights models: a single command turns downloaded weights into an OpenAI-compatible API with serious throughput. This tutorial walks you from a bare GPU machine to a tuned, monitored production endpoint, covering the knobs that matter — memory allocation, prefix caching, quantization, tensor parallelism — and how to observe the result.

Prerequisites#

• A Linux host with an NVIDIA GPU (16GB VRAM is a reasonable floor for 7–8B models; 24GB is more comfortable)
• Python 3.10+ and pip, or Docker with the NVIDIA container toolkit
• Enough disk space for weights (7–8B models are ~15–16GB in bfloat16)

Step 1: Install and launch your first server#

The fastest path is pip:

pip install vllm
vllm serve meta-llama/Llama-3-8B-Instruct

vLLM downloads the weights from Hugging Face on first launch and starts an OpenAI-compatible server at http://localhost:8000. (Note that gated models like Llama require a Hugging Face token with access granted; fully open models such as Qwen or Mistral don't.) The newer vllm serve command replaces the older python -m vllm.entrypoints.openai.api_server invocation.

Test it:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3-8B-Instruct",
    "messages": [{"role": "user", "content": "What is continuous batching?"}],
    "max_tokens": 256
  }'

Or use the OpenAI Python SDK unchanged, pointing base_url at http://localhost:8000/v1. For reproducible production deploys, prefer the official Docker image instead:

docker run --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3-8B-Instruct

You can also protect the endpoint with --api-key (or the VLLM_API_KEY environment variable) — the server then requires the key in the Authorization header, and multiple keys are accepted for rotation.

Step 2: Tune throughput#

vLLM's speed comes from two core techniques:

• Continuous batching: requests are batched at the token level rather than waiting for whole sequences to finish, so GPU utilization stays high under mixed workloads.
• PagedAttention: the KV cache is managed in non-contiguous blocks, eliminating the fragmentation that forces naive servers to over-reserve memory.

To get the most out of them, tune these flags:

A sensible tuned launch for an 8B model on a single 24GB GPU:

vllm serve meta-llama/Llama-3-8B-Instruct \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --enable-prefix-caching \
  --port 8000 --host 0.0.0.0

For 30B–70B models, spread across GPUs with tensor parallelism:

vllm serve meta-llama/Llama-2-70b-hf \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --quantization awq

Two practical notes: --max-model-len interacts directly with KV-cache sizing — long context windows eat the memory you'd otherwise use for batching. And if you hit CUDA-graph out-of-memory errors on small VRAM budgets, --enforce-eager is the escape hatch (at some performance cost).

Step 3: Quantize to fit and speed up#

Quantization shrinks weights and KV cache, which lets bigger models fit on your GPUs and increases achievable batch sizes. vLLM supports several formats; choose based on what your model publisher ships:

• AWQ (--quantization awq): 4-bit weight-only, widely supported, good accuracy retention. Use pre-quantized AWQ checkpoints (e.g. TheBloke or official AWQ releases on Hugging Face).
• GPTQ (--quantization gptq): 4-bit weight-only alternative; pick the checkpoint family your model provides.
• FP8 (--quantization fp8): supported on Hopper/Ada-class GPUs (H100, L40S); halves memory versus bf16 with minimal quality loss, and often gives the best throughput per dollar where hardware allows.

vllm serve neuralmagic/Llama-3-8B-Instruct-FP8 \
  --quantization fp8 \
  --gpu-memory-utilization 0.9

Rule of thumb: if your workload is memory-bound (large models, long contexts), quantization's biggest win is fitting more KV cache and larger batches. Always spot-check quality on your own evals before and after — benchmark numbers from model publishers don't cover your task mix.

Step 4: Monitor in production#

vLLM exposes Prometheus metrics out of the box. Depending on the version, they live on the API port (http://localhost:8000/metrics) or a dedicated metrics port — set explicitly with:

vllm serve meta-llama/Llama-3-8B-Instruct \
  --enable-metrics \
  --metrics-port 9090

The metrics to watch:

• vllm:time_to_first_token_seconds — prefill latency; your primary user-perceived number
• vllm:num_requests_running — current load on the engine
• vllm:gpu_cache_usage_perc — KV-cache pressure; sustained values near 100% mean requests are queueing
• GPU utilization and any OOM errors in the logs

A minimal production checklist: TTFT under your target for typical prompts, cache usage with headroom under peak load, GPU utilization high (not idle, not thrashing), and zero OOMs. Before going live, run a load test at expected concurrency — vLLM ships a benchmarking script (benchmarks/benchmark_serving.py in the repo) — and confirm TTFT and throughput hold up before you route real traffic.

For a production Docker deploy, add --host 0.0.0.0, a reverse proxy or ingress with TLS in front, the --api-key auth, and scrape the metrics endpoint into your existing Prometheus/Grafana stack. vLLM hosts one model per server instance, so scale horizontally behind a load balancer when one GPU isn't enough.

Takeaway#

The vLLM deployment path is short: vllm serve gets you an OpenAI-compatible endpoint in one command, and the real work is tuning — memory fraction, context length, prefix caching, quantization, and parallelism — then verifying with metrics instead of vibes. Start with a bf16 baseline, add quantization where hardware allows, watch TTFT and KV-cache pressure, and load-test before you promise anyone an SLA.