System design
Evaluation platform for 20 LLM apps
Your company runs 20 LLM applications built by different teams. Design a shared way to measure and protect their quality.
45:00
45 minutes in five phases. Talk out loud the whole time.
Ask first
Who owns each app, and what do the apps have in common?
Any regulatory requirements?
How often do teams release?
Build or buy?
Core design
A shared tracing SDK, based on OpenTelemetry, sending traces to one store.
Versioned golden datasets for each app.
An evaluator library: deterministic checks, rubric-based LLM judges and human review queues.
Eval gates in CI on every change to prompts, models or retrieval.
Online sampling, dashboards and drift alerts.
Governance reports for risk and compliance teams.
Trade-offs to raise
A central platform versus team autonomy.
Judge cost versus coverage.
Self-hosted (Langfuse, Phoenix) versus a vendor.
Curveballs
Calibrate against labeled samples, tighten the rubrics, report agreement, and keep humans in the loop for high-risk apps.
Bootstrap them from production traces plus synthetic data (EvalSeed), and make a golden set part of the definition of done.
Numbers to state
Eval cost per release = examples × judges × tokens × price.