Production RAG Architectures That Survive Their First Real Customer

The end of the RAG honeymoon

If you’ve shipped a Retrieval-Augmented Generation (RAG) demo, you’ve already felt the gap between a 30-second demo and a production system that handles 10,000 queries a day, five model versions, three retrieval sources, two languages, and a customer who is not a software engineer.

The LLM is the easy part. The retrieval, the chunking, the evaluation harness, the cost control, the fallback when a vector store is down at 2am — that’s the work. This article walks through the architecture we ship at Ideaxa for technical founders building production RAG.

What “production” actually means

Three signals tell you a RAG system has crossed from demo to production:

Latency budget is enforced, not aspirational. p95 retrieval + generation under 3 seconds, with a fallback path when the LLM provider degrades.
Evaluation runs on every commit. A frozen golden set, regression alerts, and a CI gate that blocks deploys on quality regression.
Cost is bounded. Per-tenant budgets, model-tier routing (cheap model for 80% of queries, expensive model for the 20% that need it), and prompt caching.

The chunking problem nobody wants to talk about

Most “RAG is bad” complaints are actually “the chunking is bad” complaints. Garbage in, garbage out — even when the LLM is GPT-5 or Claude Opus 5.

Three chunking strategies that work in production

Semantic chunking by section headers. For structured documents (Markdown, docs sites, Notion exports), chunk on H1/H2/H3 boundaries. Preserves context the author intended.
Sliding window with overlap. For unstructured text (PDFs, long-form articles), 512-token chunks with 64-token overlap. Standard, boring, works.
Hierarchical chunking with parent retrieval. Chunk into small pieces (256 tokens) for retrieval, but return the parent (1024 tokens) at inference. Gives the LLM the surrounding context without polluting the embedding space.

The fourth strategy — “let the embedding model figure it out” — does not work. Embedding models don’t have document structure awareness. They see tokens.

Metadata matters more than the embedding

A chunk with a clean text body but no metadata is a wasted retrieval. At minimum, every chunk needs:

source — document ID, URL, or canonical reference
section — breadcrumb path within the source
timestamp — for time-window filtering
tenant_id — for multi-tenant isolation
language — for cross-lingual retrieval

Retrieval that filters on metadata before the vector search is 10x more accurate than retrieval that doesn’t. Add the metadata, then do a small, focused similarity search. Don’t do similarity search on a million chunks and hope the right one floats up.

Hybrid search: BM25 + vectors, the only way that works

Vector search is great at semantic similarity (“what is the policy on remote work?” matches “WFH guidelines”). BM25 is great at exact term match (“kubernetes operator” matches documents containing the literal phrase).

You need both. Production RAG that uses only vector search misses 20-30% of obvious queries. Production RAG that uses only BM25 misses 30-40% of semantic queries. Hybrid search that combines both (typically with reciprocal rank fusion) gets >90% on most workloads.

Reciprocal Rank Fusion (RRF) — the boring 12 lines that matter

def rrf(ranks_lists, k=60):
    # ranks_lists: list of lists of doc_ids, one per retrieval source
    scores = {}
    for ranks in ranks_lists:
        for rank, doc_id in enumerate(ranks):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: -x[1])

Run BM25, run vector search, combine with RRF, rerank with a cross-encoder, return top-k. That pipeline is the default we ship.

The model layer: tier, route, and fall back

The single biggest cost reduction in production RAG is model tiering. Most queries don’t need the most expensive model.

Three tiers, in order

Tier 1 — small model (e.g. Haiku 4, GPT-5-mini, Llama 3.3 8B on-prem). Simple factual lookups, single-document answers, structured extraction. ~$0.001 per query. 80% of traffic.
Tier 2 — mid model (e.g. Sonnet 4.5, GPT-5). Multi-document reasoning, summarization, complex instructions. ~$0.01 per query. 15% of traffic.
Tier 3 — large model (e.g. Opus 5, o3). Open-ended reasoning, code generation, multi-step planning. ~$0.10 per query. 5% of traffic.

Route by query complexity. Use a cheap classifier (even a small LLM or a fine-tuned BERT) to predict the tier. If the classifier is wrong 10% of the time, you still save 60% on inference cost.

Prompt caching is free money

If your system prompt and retrieved context are stable across queries, every major LLM provider now supports prompt caching. Anthropic, OpenAI, and Google all charge 10% of the input price for cached tokens. For a system prompt + 2,000 tokens of retrieved context, that’s an 80% reduction on input cost for every query after the first.

Always have a fallback

Your primary LLM provider will have an outage. Probably during your biggest customer’s demo. Configure:

Primary + secondary provider, automatic failover on 5xx or 30s+ latency
Circuit breaker that doesn’t hammer a degraded provider
Cached responses for the top 1,000 most common queries (serve from cache during outage)
A 60-second “we’re experiencing issues” message instead of an error

Evaluation: the thing that separates demo from product

If you don’t have an evaluation harness, you don’t have a RAG system. You have a demo.

Build a golden set of 200+ queries

Hand-curate 200-500 real production queries, each with a “what a good answer looks like” annotation. The annotations don’t need to be perfect — they need to be consistent and recent.

Three evaluation metrics that matter

Faithfulness — is the answer supported by the retrieved context? (RAGAS, custom LLM judge, or a hand-tuned prompt)
Answer relevance — does the answer actually address the question?
Context precision — are the top-k retrieved chunks actually relevant?

Run evals on every PR

Add the eval suite to your CI. If a prompt change drops faithfulness by 3%, block the deploy. If a model upgrade drops answer relevance by 2%, block the deploy. This is the only way to ship LLM changes with confidence.

Real production numbers we see

Faithfulness: 0.92-0.96 on a well-built system, 0.7-0.85 on a typical “we just added RAG” system
Answer relevance: 0.85-0.92 for tier 1, 0.90-0.96 for tier 2/3
Context precision: 0.7-0.85 (always lower than you’d expect — that’s why reranking matters)

The architecture we ship

The reference architecture we deploy at Ideaxa for production RAG:

Document ingest — S3 + Lambda / Cloud Functions + Document AI for OCR, normalized to Markdown.
Chunking pipeline — semantic + hierarchical, with metadata enrichment.
Embedding — Voyage 3 / OpenAI text-embedding-3-large / Cohere embed-v3. Batched, with prompt caching.
Vector store — pgvector (default), Pinecone (high scale), or Weaviate (hybrid search native). pgvector is the right answer for 80% of startups.
BM25 index — OpenSearch or Elasticsearch. Kept in sync with the vector store.
Retrieval — hybrid (vector + BM25) → rerank (Cohere rerank-v3 or local cross-encoder) → top-k=8.
Generation — tier-routed, with prompt caching, with fallback.
Evaluation — RAGAS + custom LLM judge, run on every PR and nightly against live traffic.
Observability — Langfuse / Helicone / custom. Trace every retrieval + generation, with cost and latency per call.

The whole thing runs on AWS or GCP, costs $800-$4,000/month at 1M queries/month, and survives a single provider outage without a customer noticing.

That’s the bar. Anything less is a demo.