Enterprise LLM Gateway

• Limits on both requests and tokens per minute, per key, tenant and model.
• Estimate tokens before the call (input tokens plus max_tokens), reserve them, then reconcile with actual usage.
• A token bucket or sliding window in Redis, updated atomically with Lua scripts.
• Return 429 with Retry-After.

Counting requests only.

• Closed → open when errors cross a threshold in a window → half-open after a cooldown to probe → closed again on success.
• Kept per provider and model, and combined with a fallback chain.

Confusing circuit breaking with retries.

• Classify the request (embedding similarity to route examples, or a small classifier), map it to a model tier, and fall back to a default.
• Measure routing accuracy and the cost-quality trade-off.

Routing with no way to tell whether it’s right.

• Stateless pods behind a load balancer, autoscaled on concurrency.
• Redis Cluster for counters; local token buckets synced periodically to cut Redis round trips.
• Long-lived streaming connections, regional quotas, and metric cardinality kept under control.

One Redis instance and “add more pods.”

• Be honest: control and learning, plus custom PII and budget logic.
• In an enterprise you’d weigh buy versus build: LiteLLM proxy, Kong AI Gateway, Apigee, or the AI gateway policies in Azure API Management.

Not knowing these alternatives exist.

• Proxy server-sent events chunk by chunk, and read the usage block in the final chunk (stream_options include_usage) or count with a tokenizer.
• Cancel the upstream call when the client disconnects.

Buffering the whole response before sending it.