System design
Move from API models to self-hosted
Your GenAI app’s API bill is growing fast, and leadership wants to move to self-hosted open models. Plan the move.
45:00
45 minutes in five phases. Talk out loud the whole time.
Ask first
Why move: cost, data residency, latency or control?
Is traffic steady or spiky?
What’s the quality bar, and how is it measured today?
Does the team have GPU operations experience?
Core design
Baseline today’s quality, latency and cost on a golden set.
Shortlist open models, and fine-tune or distill if needed.
Serve with vLLM or SGLang on GKE or EKS, autoscaling on queue depth and GPU utilization.
Shadow traffic first, then a canary, then a gradual rollout through the gateway.
Keep the API as a fallback and for overflow; monitor quality, latency and cost per request.
Trade-offs to raise
Self-hosting is cheaper only at high, steady GPU utilization.
Quantization saves memory but can cost quality.
Operational burden versus control.
Curveballs
Run a hybrid: self-host the baseline and overflow to the API. Scaling GPUs up from zero is slow because models take time to load.
Decide whether that’s within tolerance; fine-tune, or route hard queries to the API model.
Numbers to state
Self-hosted cost per 1M tokens = GPU cost per hour ÷ tokens per hour at real utilization × 1M.
A 7B model in FP16 needs about 14 GB for the weights alone.