Multi-tenant LLM gateway

Design a central gateway that every team in a large company uses to call LLMs from several providers.

45:00

45 minutes in five phases. Talk out loud the whole time.

Ask first

How many teams, providers and requests per second?

Is streaming needed?

What are the rules on PII, logging and data residency?

Budgets per team? Any self-hosted models?

Authentication with API keys or OAuth, and tenant identification.

Policy checks: model allowlists, quotas and budgets.

Pre-processing: PII masking, prompt validation and injection screening.

Routing by task, cost or health to provider adapters behind one OpenAI-compatible interface.

Resilience: timeouts, retries with jitter, circuit breakers and fallback chains.

Post-processing: output guardrails and unmasking; usage metering, traces and audit logs; optional caching.

Central control versus added latency. Keep gateway overhead in the low tens of milliseconds.

Semantic cache hit rate versus the risk of a wrong cached answer.

Logging prompts for debugging versus privacy.

Health checks trip the breaker, and traffic fails over to another region or provider that still meets data-residency rules.

Reserve tokens before the call. Soft limits warn and hard limits stop new requests, never a response already in progress.

Stateless pods, Redis Cluster for counters, local token buckets with periodic sync, and regional deployments.

A gateway overhead budget, such as under 20 ms at p95.

Redis operations per request: two to four.