RAG in production: what breaks past the demo

A retrieval-augmented generation pipeline is trivial to demo and stubborn to operate. The first weekend you hit ninety percent on a curated eval set. Six weeks later, real users are asking questions your retriever never anticipated, your context window is a landfill of adjacent-but-wrong chunks, and your answers regress in ways nobody can reproduce.

This is a working note on the shape of a RAG system that behaves like software, not like a science project. Nothing here is novel; everything here is the difference between a demo and a system you would put your name on.

Three forces that break every RAG

There are three failure modes I see in almost every stalled deployment: query drift, retrieval noise, and untracked context. They cascade — a small failure in one amplifies the others — so you have to design for all three from day one.

Query drift

Users do not ask the questions your golden dataset expected. They paraphrase, elide, and reference prior turns. If your embedding of the raw query is your only key into the index, half of your traffic falls off a cliff the moment usage broadens.

Retrieval noise

Cosine similarity is not a truth function. Chunks that mention the right words but the wrong concept are indistinguishable to a nearest-neighbor lookup. Rerankers help. Structured metadata helps more.

Untracked context

The moment two systems assemble the prompt — one for retrieval, one for chat history, one for tools — you lose the ability to answer 'why did the model say that?'. Prompts are code. Log them.

A blueprint that survives contact

The pattern I now default to has five layers:

1. Query rewriting. A tiny model normalizes, decontextualizes, and multiplies the query before it ever touches the vector store.
2. Hybrid retrieval. BM25 for lexical anchors, embeddings for semantic reach, metadata filters for tenancy and freshness.
3. Rerank + budget. A cross-encoder narrows the shortlist, then a token-budget planner packs the winners without spilling.
4. Grounded synthesis. The generator sees the passages, their citations, and an explicit instruction to abstain when coverage is thin.
5. Full traceability. Every step writes to a run log — inputs, chosen chunks, prompts, tokens, latency, cost.

Evaluation is a product surface

Treat your eval harness like a first-class product. It needs its own UI, its own regression gate, and its own owner. When a new prompt or a new index ships, the harness runs three flavors of check: retrieval hit rate, groundedness, and answer preference. If any of the three regresses beyond a threshold, the deploy is blocked.

type EvalCheck = {
  id: string;
  input: string;
  goldPassages: string[];
  goldAnswer?: string;
  tags: string[];
};

async function runEval(checks: EvalCheck[], pipeline: RagPipeline) {
  return Promise.all(checks.map(async (c) => {
    const trace = await pipeline.answer(c.input);
    return {
      id: c.id,
      hit: passageRecall(trace.retrieved, c.goldPassages),
      grounded: groundednessScore(trace.answer, trace.retrieved),
      preference: c.goldAnswer ? await judge(trace.answer, c.goldAnswer) : null,
    };
  }));
}

Signals that predict trouble

Long before user complaints arrive, these signals tell you the system is fraying:

• Retrieval hit rate drops on a specific tag or tenant.
• Groundedness scores fall while answer confidence stays high — the classic hallucination signature.
• Token cost per resolved query climbs while resolution rate stays flat.
• Latency P95 walks up 100 ms per week without an obvious cause.

Shipping a model is easy. Shipping a system that stays honest is the whole job.

Where to start

If you already have a demo, do not add features. Add observability. Log every retrieval, every rerank, every generation, and every user thumb. Give yourself a week of production data before you touch the pipeline again — and then let the data pick your next problem.