A RAG system that answers every question is not a good RAG system — it is a system that has not yet been asked a question its corpus cannot answer. The failure mode that matters at scale is not “the model is not smart enough.” It is “the model was never told that not answering is an option.”

Situation

RAG corpora keep growing — from a few thousand internal documents toward multi-million-document scale — and the standard response to hallucination has been to reach for a better generator model. Larger models are more fluent, but fluency is not grounding. A bigger model still produces a confident, well-formed sentence when retrieval comes back empty or ambiguous, because producing fluent text is what generation does regardless of whether the evidence supports it.

The Problem

Most RAG pipelines have exactly one failure mode when evidence is weak: guess anyway, but say it more articulately. There is no separate path for “I don’t actually know this.” Citations, when present, are usually decorative — the model is asked to cite sources, but nothing checks whether the cited passage actually supports the sentence attached to it, or whether the citation exists at all.

That gap has three concrete shapes:

  • A model can cite a real passage and still misread what it says.
  • A model can cite a passage id that does not exist in the retrieved set — an invented citation wearing the appearance of evidence.
  • A model can answer confidently on a question the corpus was never able to answer, because refusing was never presented as a valid output.

The question that actually matters for a production RAG system is not “how good is the retriever” or “how good is the model.” It is: when the evidence is missing or weak, does the system have a designed, measurable way to say so — or does it just guess more fluently than a smaller model would?

The Retrieve-Constrain-Verify-Abstain Pattern

Fareed Khan’s open-source project and accompanying write-up, “Building a RAG Pipeline for 10M+ Documents With Near-Zero Hallucination” (published in Level Up Coding, June 2026), is one of the more complete public treatments of this problem I’ve seen — full code, full evaluation numbers, and an explicit accounting of where the approach still falls short. The architecture is four control layers stacked on top of an ordinary open-weight model, not a smarter model on its own:

  1. Retrieve the right evidence — hybrid dense-plus-BM25 search over contextualized chunks, fused and reranked.
  2. Constrain generation — answer only from the retrieved context, cite a passage id on every sentence, or emit an explicit abstain token.
  3. Verify every atomic claim — split the answer into individually checkable claims and score each against its cited text with a faithfulness judge.
  4. Abstain — fold the routing signal, the generation signal, and the verification signal into one decision, and refuse when support is not there.

Those four layers are wired into a self-correcting loop, built with LangGraph, that behaves like Corrective RAG (CRAG): grade the retrieved evidence before generating anything, and if it’s weak, refine the query and re-retrieve — up to a bounded number of hops — rather than generating from thin context and hoping verification catches it after the fact.

flowchart TD
    Q[user question] --> R[route the question]
    R --> H[hybrid retrieve — dense and BM25]
    H --> F[fuse with reciprocal rank fusion]
    F --> RR[rerank top candidates]
    RR --> G[grade evidence sufficiency]
    G -->|strong| GEN[generate with citations]
    G -->|weak — hops remain| RE[refine query and re-retrieve]
    RE --> H
    G -->|weak — hops exhausted| AB[abstain]
    GEN --> V[verify every atomic claim]
    V -->|all claims supported| ANS[answer with citations]
    V -->|any claim unsupported| AB

The design principle underneath all four layers: the system has exactly one safe failure mode, and it’s abstention, not a fluent guess.

In Practice

Khan’s implementation is concrete enough to check, and the numbers are reported honestly, including where the approach is weak. The full pipeline, code, and run outputs are public on GitHub, so this is CARL category A — a documented, cited, reproducible individual project, not a company’s internal decision, and not something I’ve independently re-run.

Retrieval. Passages are cleaned (NFKC normalization, MinHash LSH near-duplicate removal) and chunked sentence-aware — packing whole sentences to a 256-token budget with 32-token overlap, counted with the generator’s own tokenizer, specifically to avoid silently truncating the one sentence that holds the answer. Each chunk gets a one-line, LLM-generated situating sentence prepended before indexing (a local-model take on Anthropic’s contextual retrieval idea), which the write-up credits as “most of why recall ends up so high.” Every chunk is indexed twice: as a dense vector in LanceDB (embedded, on-disk, no server — chosen specifically because it doesn’t require holding the whole index in RAM) and as a sparse BM25 posting via bm25s. The two ranked lists are combined with Reciprocal Rank Fusion (score = sum of 1/(k + rank + 1) across lists, k=60) — which sidesteps the fact that a cosine similarity and a BM25 score aren’t on comparable scales by ignoring the scores entirely and using only rank position. 150 fused candidates get reranked down to 20 by a Qwen3-Reranker-4B model reading each (query, passage) pair and scoring P(yes) from next-token logits. On the full evaluation set, this retrieval stack reaches 0.97 context recall.

Constraint and verification. Generation (Qwen3-32B, served locally via vLLM, temperature 0) is instructed to answer only from the numbered context, cite a passage id on every sentence, or emit an explicit abstain token — and after generation, any citation that doesn’t match a real retrieved chunk id is stripped from the text before a user ever sees it. The verification gate then splits the drafted answer into atomic claims and scores each one against its cited passages with a faithfulness judge (the same local 32B model, prompted as a strict fact-checker). Critically, the gate reports the minimum claim score, not the average — a long answer that’s 80% grounded and smuggles in one invented sentence still fails, because “an answer is only as trustworthy as its least supported sentence.”

The measured result. On a 200-question golden set (100 answerable HotpotQA questions, 100 unanswerable — a mix of SQuAD v2 impossible questions and hand-written false-premise questions like “Which programming language did Isaac Newton invent in 1700?”), the system produced a 2x2 confusion matrix of [[46, 54], [2, 98]] — rows answerable/unanswerable, columns answered/abstained. Read the second row: of 100 unanswerable questions, the system abstained on 98 and hallucinated an answer on only 2 — a 2% hallucination rate, not zero, which the author states plainly (“literal zero is not achievable from a generative model”). The cost shows in the first row: only 46 of 100 answerable questions were actually answered; the rest were abstained too, at the threshold chosen to hold hallucination under a 5% budget. On the answered subset: 0.908 faithfulness, 0.817 answer relevancy. And the faithfulness judge itself — the component the entire gate depends on — scores only 0.702 AUROC against HaluBench, a human-labeled faithful/hallucinated benchmark, which the author calls “good but not great” and names as the single highest-leverage thing to improve next.

The scale claim, separately. The “10M+ documents” part of the title is a different benchmark than the quality numbers above — this distinction matters and is easy to miss on a skim. The quality pipeline was evaluated on a 20,007-passage corpus. The 10M-vector test used synthetic 1024-dimensional random unit vectors in a LanceDB IVF_PQ approximate index, purely to measure latency and disk growth, not retrieval quality (recall@10 on random vectors is reported near 0.10-0.13, which the author explicitly flags as meaningless — there’s nothing real to find). What that benchmark does show: p95 query latency moved from 10.59ms at 100K vectors to 18.48ms at 10M — roughly a 100x growth in index size for less than 2x growth in latency, because an IVF_PQ index searches a fixed number of partitions rather than scanning linearly. Disk grew linearly and predictably, from 0.39GB to 38.8GB. A linear extrapolation to 100M projects 77.58ms p95 and 388GB — plausible on a single machine’s NVMe, but a projection, not a measurement. Separately, per-stage latency on the actual quality-pipeline runs shows retrieval (embedding + dense search + sparse search + cross-encoder reranking over 150 candidates) as the dominant cost at a mean of 4.2s — not the vector index itself, which stays cheap at any scale tested.

Where It Breaks

Design choiceWhat it costsWhy it’s still probably right
Threshold tuned for ≤5% hallucinationOnly 46% of answerable questions get answered — most of the system’s output is refusalThe alternative is a confident wrong answer; coverage is a dial, not a fixed cost, and can be set per corpus
Claim-level gate, minimum not averageSlower — every answer requires extracting and scoring N claims, not one scoreAn answer-level average would let one hallucinated sentence hide inside a mostly-true paragraph
Faithfulness judge is the same local 32B modelGate quality is capped at 0.702 AUROC — roughly a coin flip plus a thirdIt’s still the highest-leverage single upgrade path; the architecture doesn’t need to change to swap in a stronger judge
All models are one local Qwen3 family, no external API callsQuality ceiling is whatever that model family can do; no frontier-model fallbackThe entire premise is a private corpus where no document or query can leave the machine — that constraint is the point, not an oversight
10M-vector benchmark uses synthetic vectorsDoesn’t prove recall holds at real 10M-document scale — only that latency and disk doIt isolates the index-scaling question from the retrieval-quality question, which is the right way to test two different claims separately

What to Do Next

  • Problem: Most RAG systems have no designed failure mode for “the evidence isn’t there” — they guess fluently instead of refusing, and a citation on the answer doesn’t mean the citation was checked.
  • Solution: Wrap an ordinary model in four explicit gates — hybrid retrieval with fusion and reranking, generation constrained to cited passages only, claim-level faithfulness verification using the weakest claim, and an abstention policy that folds all three signals into one decision.
  • Proof: On a balanced answerable/unanswerable golden set, this pattern produces a measurable, tunable hallucination rate (2% in Khan’s run) instead of an unmeasured one, at an explicit and visible coverage cost (46% here) — both numbers you can move by adjusting the threshold, not by hoping.
  • Action: Before adding retrieval features to reduce hallucination, check whether your pipeline has any of these three things at all: citation validation that strips invented citations, a claim-level (not answer-level) faithfulness check, and an abstain path that a router or verifier can actually reach. If none exist, that’s the gap to close first — regardless of corpus size.

Sources