RAG quality is capped by retrieval quality. If the search layer cannot find the right products, listings, policies, documents, or runbooks, the LLM will only make the missing evidence harder to see.

Situation

Natural language search is spreading into product catalogs, travel search, real estate search, food ordering, support portals, policy systems, and internal engineering knowledge bases.

The common first version is simple: embed documents, retrieve the top chunks, and send them to an LLM. That works for demos because the corpus is small, the examples are curated, and the user expects an answer instead of a ranked search experience.

Production search is less forgiving. Users type exact names, misspell terms, mix constraints with intent, expect filters to hold, and notice when availability or price is wrong. They also expect a result page to be stable enough to compare, sort, and trust.

That is why hybrid search should come before RAG in most serious search architectures.

The Problem

Pure vector retrieval has predictable blind spots:

  • It can miss exact identifiers.
  • It can blur legal or medical terms that should remain exact.
  • It may retrieve semantically similar but unavailable items.
  • It can underperform when filters are selective.
  • It does not provide faceting, spelling correction, synonym control, ranking analytics, or explainable field relevance by itself.

Pure lexical search has its own blind spots:

  • It may miss paraphrases.
  • It may fail when users describe intent instead of terms.
  • It can overweight rare tokens that are not actually important.
  • It requires careful analyzers and field modeling.

RAG does not remove either problem. It adds answer generation after retrieval. If the retrieval system returns the wrong candidate set, the LLM has no reliable way to recover.

The production baseline should be:

Lexical retrieval plus semantic retrieval plus hard filters plus rank fusion plus business ranking plus optional reranking.

Only after that should the system decide whether an LLM answer is needed.

The Hybrid Retrieval Pipeline

flowchart TD
    Q[User query] --> P[Query understanding]
    P --> L[BM25 candidates]
    P --> V[Vector candidates]
    P --> F[Hard filters]
    F --> L
    F --> V
    L --> C[Candidate pool]
    V --> C
    C --> X[Rank fusion]
    X --> B[Business ranking]
    B --> R[Reranker]
    R --> N{Needs answer}
    N -->|No| S[Results page]
    N -->|Yes| G[RAG answer]

BM25 or full-text search should handle exact words, phrases, identifiers, and field-aware relevance. Vector search should handle semantic similarity and discovery. Filters should enforce non-negotiable constraints: tenant access, deletion state, geography, price range, inventory, room availability, policy version, region, and compliance state.

Fusion should combine candidates without pretending raw scores are comparable. BM25 scores and vector similarity scores come from different systems. A rank-based method such as reciprocal rank fusion is often easier to reason about than raw score mixing. Score normalization can also work, but it must be tested with real query sets.

Business ranking is separate from retrieval. A product or listing may match the query but still rank lower because it is unavailable, stale, low quality, outside the user’s constraints, or less useful for the current session.

Reranking should be bounded. It is expensive to rerank thousands of candidates. A good system retrieves broadly enough, fuses candidates, then reranks a controlled set.

In Practice

The documented pattern in OpenSearch is a search pipeline: run keyword and semantic clauses, normalize scores, and combine them. That matters because it forces the team to acknowledge that hybrid retrieval is a pipeline, not one magic query.

The documented pattern in PostgreSQL plus pgvector is more manual but useful for smaller systems. PostgreSQL full-text search can produce lexical candidates. pgvector can produce semantic candidates. SQL can apply tenant and lifecycle filters. Application code can fuse candidates and log which path contributed each result.

The documented pattern in Weaviate, Qdrant, and Pinecone is that sparse or keyword-style signals remain important even inside AI-native retrieval systems. Dense vectors are not enough for every query. The exact implementation differs, but the architectural lesson is stable: production retrieval usually needs more than one signal.

For industry systems, the consequences are straightforward.

Retail catalog search needs exact product names, categories, attributes, price, stock, brand, reviews, personalization, and semantic discovery. Travel search needs destination, dates, guests, room availability, amenities, cancellation rules, taxes, fees, distance, and user intent. Marketplace search needs geography, availability, seller or listing state, price, freshness, trust, and semantic matching.

The LLM can help interpret “quiet place near downtown with breakfast and parking.” It should not decide the final price, inventory, compliance eligibility, or access rights.

Where It Breaks

Failure modeSymptomMitigation
Vector-only retrievalExact names, IDs, and regulated terms disappearAdd lexical retrieval
Keyword-only retrievalNatural-language intent is missedAdd semantic retrieval
Soft filtersResults violate price, access, inventory, or region constraintsTreat constraints as hard filters
Raw score mixingOne retrieval path dominates unpredictablyUse normalization or rank fusion
No query classesOne ranking recipe handles every query badlySeparate exact lookup, discovery, and troubleshooting
Reranker overuseCost and latency rise for simple queriesRoute by query class
RAG too earlyAnswers hide retrieval defectsinspect candidates before generation

What to Do Next

  • Problem: Decide which user queries require exact matching, semantic recall, hard filtering, ranked results, or generated explanation.
  • Solution: Build a hybrid candidate pipeline before exposing LLM answers as the main product surface.
  • Proof: Measure lexical-only, vector-only, hybrid, and reranked results against a fixed query set with expected documents and negative cases.
  • Action: Log query class, candidate source, filter selectivity, fusion rank, reranker rank, latency, empty-result rate, and no-click rate.

The safest production path is not to choose between BM25 and vectors. It is to make both accountable.

Hybrid search is the retrieval baseline. RAG is an answer layer. Agents are a workflow layer. GraphRAG is a relationship layer. Collapsing those layers into “AI search” is how teams lose control of correctness.

Sources