vllm-serving-setup

When to Use

• When deploying a specific model (e.g., Llama 3.3 70B, Qwen3 32B, DeepSeek-V3) to EKS with vLLM
• When tuning the throughput and latency of an existing vLLM deployment
• When putting Multi-LoRA adapters on a single base model
• When enabling new features in v0.18.2 such as FP8 KV Cache and Chunked Prefill

When NOT to Use

• Large-scale traffic requiring distributed inference (Disaggregated Prefill/Decode) → llm-d skill usage recommended
• Training pipeline → SageMaker / KubeRay / SkyPilot domain
• When MoE-specific optimization is needed → refer to Expert Parallel dedicated configuration

Preconditions

• GPU NodePool, GPU Operator, DCGM operating normally on the EKS cluster (agentic-eks-bootstrap complete)
• Hugging Face access token (stored in Secrets Manager) or S3 model weights copied
• Prometheus + OTel Collector deployed

Procedure

Step 1. GPU memory sizing

필요 GPU 메모리 = 모델 가중치 + 비torch 메모리
                  + PyTorch 활성화 피크 메모리
                  + (배치당 KV 캐시 × 배치 크기)

• Llama 3.3 70B FP16 → weights 140GB + KV cache ~40GB + overhead 20GB ≈ 200GB
• Single H100 80GB not possible → TP=4 (50GB per GPU)
• With INT4 quantization, 35GB → single A100 80GB or H100 possible

Step 2. Selecting a parallelization strategy

• TP (Tensor Parallel): distribute layer parameters across GPUs within the same node
• PP (Pipeline Parallel): distribute layer groups across nodes (multi-node)
• EP (Expert Parallel): exclusively for MoE models
• DP (Data Parallel): replica scaling, combined with HPA

Step 3. Writing Helm values

model:
  name: meta-llama/Llama-3.3-70B-Instruct
  dtype: bfloat16
vllm:
  image: vllm/vllm-openai:v0.18.2
  extraArgs:
    - --tensor-parallel-size=4
    - --gpu-memory-utilization=0.92
    - --max-model-len=32768
    - --enable-chunked-prefill
    - --enable-prefix-caching
    - --kv-cache-dtype=fp8
    - --enable-lora
    - --max-loras=8
    - --otlp-traces-endpoint=http://otel-collector.observability.svc:4317
resources:
  limits:
    nvidia.com/gpu: 4
nodeSelector:
  karpenter.sh/nodepool: gpu-inference-pool
tolerations:
  - key: nvidia.com/gpu
    operator: Exists

Step 4. Deployment and verification

helm upgrade --install vllm-llama3 vllm-project/vllm \
  --namespace inference --create-namespace \
  -f values.yaml --version 0.18.2

kubectl -n inference rollout status deployment/vllm-llama3 --timeout=10m
kubectl -n inference port-forward svc/vllm-llama3 8000:8000 &
curl -s http://localhost:8000/v1/models | jq
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.3-70B-Instruct","messages":[{"role":"user","content":"hello"}]}'

Step 5. HPA + PodMonitor

• HPA: vllm:num_requests_running or DCGM_FI_DEV_GPU_UTIL based
• PodMonitor: /metrics endpoint (prometheus_client)
• Verify that OTel traces are being sent to Langfuse

Step 6. Benchmark

python benchmarks/benchmark_serving.py \
  --backend vllm \
  --model meta-llama/Llama-3.3-70B-Instruct \
  --num-prompts 200 \
  --request-rate 10

• Capture p50/p95/p99 latency, throughput (tokens/s), GPU util

Good Examples

• Llama 3.3 70B: TP=4, FP8 KV Cache, Chunked Prefill on → 30% throughput improvement
• Qwen3 32B: TP=2, Prefix Caching enabled → 400%+ improvement on code benchmark
• 16 Multi-LoRAs: single base + adapter hot swap, per-customer tenant separation

Bad Examples (prohibited)

• gpu-memory-utilization=0.98 — OOM risk, 0.92 or lower recommended
• --max-model-len Allowing long prompts without arbitrary reduction → KV cache explosion
• TP setting does not match number of GPUs (e.g., TP=4 but requesting 3 GPUs) — boot failure
• Using old vLLM v0.6.x — Chunked Prefill, FP8 KV not supported

References

• vLLM model serving (community resource)
• llm-d distributed inference (community resource)
• vLLM official documentation
• vLLM Production Stack
• PagedAttention SOSP 2023
• Plugin internals: ../../references/vllm-performance-tuning.md