System design
Enterprise RAG over 10 million documents
Design a question-answering assistant over 10 million internal documents for 50,000 employees. Answers must cite sources and respect document permissions.
45:00
45 minutes in five phases. Talk out loud the whole time.
Ask first
Which document types, and how often do they change?
How many users and queries per second at peak?
What’s the latency target, and is streaming acceptable?
Where do permissions live: Active Directory, Okta, SharePoint?
Languages, data residency and retention rules?
What does “good enough” accuracy mean to the business?
Core design
Connectors pull documents and permission lists; changes arrive as events.
Layout-aware parsing and OCR, structure-aware chunking, and a short contextual summary per chunk.
Embeddings plus BM25 in a hybrid index, with permission metadata on every chunk.
Query path: authenticate, rewrite the query, retrieve with a permission filter, fuse, rerank the top 50 down to 8.
Generate with citations, run output guardrails, and stream the answer.
Trace every request; evaluate offline on a golden set and online on sampled traffic.
Trade-offs to raise
One index with filters versus shards per department or region.
Embedding size versus storage cost, and quantizing vectors.
Freshness versus indexing cost.
A bigger reranker versus latency.
Curveballs
Sync permissions separately from content, on a short refresh interval, and check the identity provider at query time for sensitive collections.
Build a new versioned index in parallel, backfill it, compare on the golden set, switch traffic, then retire the old index.
Retrieval-score thresholds, explicit permission to abstain in the prompt, and distractor questions in the evals.
Cache frequent questions, route simple ones to a smaller model, trim context and cap output tokens.
Numbers to state
10M documents × about 20 chunks ≈ 200M vectors.
1,024 dimensions × 4 bytes ≈ 4 KB per vector, or about 800 GB raw before quantization.
A latency budget, such as under 2 s to first token at p95.