Sub-Second Global Search & Discovery: Blending Vector Search with Keyword Filtering for Catalogs

For CTOs and engineering leaders running a catalog of 500,000 to 50 million SKUs, search is the most revenue-sensitive system you own, and usually the least understood. It sits between intent and inventory. Every extra 300 milliseconds and every "no results" page is a quiet tax on conversion.

Most teams arrive at the same fork in the road. Classic keyword search (BM25 over an inverted index) is fast, predictable and excellent at exact tokens such as SKU codes, brand names and model numbers. It fails on vocabulary mismatch: a shopper types "waterproof jacket for hiking in drizzle" and the catalog says "weatherproof shell, lightweight". Pure vector search fixes the semantics but introduces new problems: it blurs exact identifiers, it is awkward to combine with hard filters (size, stock, region, price), and it can return confident nonsense.

The answer in production is almost never one or the other. It is a hybrid retrieval pipeline: lexical and semantic candidates retrieved in parallel, constrained by the same structured filters, fused with a rank-based algorithm, and optionally reranked. This article walks through the architecture, the code, the failure modes and the decisions that matter. All numbers below are illustrative worked examples with stated assumptions, not measurements from a specific deployment.

1. Why Neither Retrieval Mode Is Enough#

2. The Latency Budget: What "Sub-Second" Actually Means#

"Sub-second" is the end-to-end figure the shopper feels. A useful working target is a p95 of roughly 400 ms from request to rendered results, leaving headroom for the network and the front end. Here is a typical illustrative budget for a regionally deployed stack:

3. Reference Architecture#

The flow is: edge cache lookup, query understanding (spelling, language, filter extraction), a cached query embedding, then BM25 and filtered ANN retrieval in parallel (top 100 each), RRF fusion, optional reranking of the top 50, and finally hydration of price, stock and images.

Two design rules keep this honest. Both retrievers receive the same filter set, so fusion never promotes an item that violates a hard constraint. And both run concurrently, so total retrieval latency is the slower of the two, not the sum.

4. Indexing: One Document, Two Representations#

Each product is indexed with a lexical representation (title, brand, attributes, synonyms) and a dense vector. What you embed matters more than which model you pick. A reliable pattern is to embed a compact, templated string instead of raw marketing copy:

400 font-semibold">class="text-emerald-300">"{brand} {title}. Category: {category_path}. Key attributes: {top_attributes}. {short_description}"

Option A: PostgreSQL with pgvector#

If your catalog is under roughly 5 to 10 million items and you already run Postgres, pgvector keeps everything transactional and removes a system from your estate.

400 font-semibold">CREATE EXTENSION IF NOT EXISTS vector;

400 font-semibold">CREATE 400 font-semibold">TABLE products (
  id            bigint PRIMARY KEY,
  region        text        NOT NULL,
  category_id   int         NOT NULL,
  brand         text,
  price_cents   int         NOT NULL,
  in_stock      400">boolean     NOT NULL,
  title         text        NOT NULL,
  description   text,
  tsv           tsvector GENERATED ALWAYS AS (
                  setweight(to_tsvector(400 font-semibold">class="text-emerald-300">'english', coalesce(title,400 font-semibold">class="text-emerald-300">'')), 400 font-semibold">class="text-emerald-300">'A') ||
                  setweight(to_tsvector(400 font-semibold">class="text-emerald-300">'english', coalesce(brand,400 font-semibold">class="text-emerald-300">'')), 400 font-semibold">class="text-emerald-300">'B') ||
                  setweight(to_tsvector(400 font-semibold">class="text-emerald-300">'english', coalesce(description,400 font-semibold">class="text-emerald-300">'')), 400 font-semibold">class="text-emerald-300">'D')
                ) STORED,
  embedding     vector(1024) NOT NULL
);

400 font-semibold">CREATE 400 font-semibold">INDEX products_tsv_idx ON products USING gin (tsv);
400 font-semibold">CREATE 400 font-semibold">INDEX products_hnsw_idx ON products
  USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 128);
400 font-semibold">CREATE 400 font-semibold">INDEX products_filter_idx ON products (region, category_id, in_stock);

Note that Postgres ts_rank is not true BM25. It is adequate for many catalogs, but if lexical relevance tuning is central to your business, a dedicated engine (below) gives you real BM25, analyzers and synonym handling.

Option B: OpenSearch (or Elasticsearch) for both signals#

Beyond the multi-million scale, or when you need rich analyzers, faceting and horizontal sharding, a search engine holds both representations in one index. In OpenSearch, the Lucene engine supports efficient filtering inside the k-NN search:

PUT /products
{
  400 font-semibold">class="text-emerald-300">"settings": { 400 font-semibold">class="text-emerald-300">"index": { 400 font-semibold">class="text-emerald-300">"knn": 400">true } },
  400 font-semibold">class="text-emerald-300">"mappings": {
    400 font-semibold">class="text-emerald-300">"properties": {
      400 font-semibold">class="text-emerald-300">"title":       { 400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"text", 400 font-semibold">class="text-emerald-300">"analyzer": 400 font-semibold">class="text-emerald-300">"english" },
      400 font-semibold">class="text-emerald-300">"brand":       { 400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"keyword" },
      400 font-semibold">class="text-emerald-300">"region":      { 400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"keyword" },
      400 font-semibold">class="text-emerald-300">"category_id": { 400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"integer" },
      400 font-semibold">class="text-emerald-300">"in_stock":    { 400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"400">boolean" },
      400 font-semibold">class="text-emerald-300">"price_cents": { 400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"integer" },
      400 font-semibold">class="text-emerald-300">"embedding": {
        400 font-semibold">class="text-emerald-300">"400 font-semibold">type": 400 font-semibold">class="text-emerald-300">"knn_vector",
        400 font-semibold">class="text-emerald-300">"dimension": 1024,
        400 font-semibold">class="text-emerald-300">"method": {
          400 font-semibold">class="text-emerald-300">"name": 400 font-semibold">class="text-emerald-300">"hnsw",
          400 font-semibold">class="text-emerald-300">"engine": 400 font-semibold">class="text-emerald-300">"lucene",
          400 font-semibold">class="text-emerald-300">"space_type": 400 font-semibold">class="text-emerald-300">"cosinesimil",
          400 font-semibold">class="text-emerald-300">"parameters": { 400 font-semibold">class="text-emerald-300">"m": 16, 400 font-semibold">class="text-emerald-300">"ef_construction": 128 }
        }
      }
    }
  }
}

5. The Hard Part: Filtering Approximate Nearest Neighbour Search#

This is where hybrid projects succeed or fail. An HNSW graph is built over all vectors. If you search it first and filter afterwards (post-filtering), a selective filter such as "region = DE AND in_stock AND size = 42" can eliminate most of the top-k, leaving you with 3 results out of the 100 you asked for. If you filter first and scan (pre-filtering by brute force), recall is perfect but latency scales with the filtered set.

Practical approaches, in order of preference:

1. Engine-native filtered ANN. OpenSearch Lucene k-NN applies the filter during graph traversal and automatically falls back to exact search when the filtered set is small. Use this by default.
2. pgvector iterative scans. From pgvector 0.8, hnsw.iterative_scan keeps scanning the graph until enough rows pass the WHERE clause (bounded by hnsw.max_scan_tuples).
3. Partitioning for low-cardinality, high-impact filters. Partition or shard by region or tenant so the graph itself is already scoped.
4. Over-fetch and cap. If you must post-filter, retrieve 5 to 10 times your target and enforce a maximum so tail latency stays bounded.

SET hnsw.ef_search = 80;
SET hnsw.iterative_scan = relaxed_order;   -- pgvector 0.8+
SET hnsw.max_scan_tuples = 20000;

400 font-semibold">SELECT id, embedding <=> $1 AS distance
400 font-semibold">FROM products
400 font-semibold">WHERE region = 400 font-semibold">class="text-emerald-300">'DE' AND in_stock AND category_id = ANY($2)
400 font-semibold">ORDER BY embedding <=> $1
LIMIT 100;

And the equivalent filtered query in OpenSearch:

POST /products/_search
{
  400 font-semibold">class="text-emerald-300">"size": 100,
  400 font-semibold">class="text-emerald-300">"query": {
    400 font-semibold">class="text-emerald-300">"knn": {
      400 font-semibold">class="text-emerald-300">"embedding": {
        400 font-semibold">class="text-emerald-300">"vector": [0.012, -0.044, 400 font-semibold">class="text-emerald-300">"..."],
        400 font-semibold">class="text-emerald-300">"k": 100,
        400 font-semibold">class="text-emerald-300">"filter": {
          400 font-semibold">class="text-emerald-300">"bool": {
            400 font-semibold">class="text-emerald-300">"filter": [
              { 400 font-semibold">class="text-emerald-300">"term":  { 400 font-semibold">class="text-emerald-300">"region": 400 font-semibold">class="text-emerald-300">"DE" } },
              { 400 font-semibold">class="text-emerald-300">"term":  { 400 font-semibold">class="text-emerald-300">"in_stock": 400">true } },
              { 400 font-semibold">class="text-emerald-300">"terms": { 400 font-semibold">class="text-emerald-300">"category_id": [210, 214] } }
            ]
          }
        }
      }
    }
  }
}

Always track recall against an exact-search baseline on a sample of real queries. A faster index that quietly loses 15% of the true neighbours is a regression you will not see in latency dashboards.

6. Fusion: Why Reciprocal Rank Fusion Is the Default#

BM25 scores are unbounded and corpus-dependent; cosine similarities live in a narrow band. Adding raw scores together is meaningless, and normalising them (min-max, z-score) is fragile because each query has a different score distribution. Reciprocal Rank Fusion (RRF) sidesteps the problem by using only ranks:

RRF(d) = sum over retrievers r of  1 / (k + rank_r(d))      (k is commonly 60)

A document ranked high by both lists wins; a document that only one retriever loves still gets a fair shot. It needs no training data and no per-query tuning, which is why it is the sensible default.

400 font-semibold">from collections 400 font-semibold">import defaultdict

400 font-semibold">def rrf_fuse(result_lists, k=60, weights=None, top_n=50):
    400 font-semibold">class="text-emerald-300">""400 font-semibold">class="text-emerald-300">"result_lists: list of ranked lists of doc ids (best first)."400 font-semibold">class="text-emerald-300">""
    weights = weights or [1.0] * len(result_lists)
    scores = defaultdict(float)
    400 font-semibold">for w, ranked in zip(weights, result_lists):
        400 font-semibold">for rank, doc_id in enumerate(ranked, start=1):
            scores[doc_id] += w * (1.0 / (k + rank))
    400 font-semibold">return sorted(scores, key=scores.get, reverse=True)[:top_n]

bm25_ids   = [101, 205, 330, 412, 518]
vector_ids = [330, 777, 101, 640, 205]
print(rrf_fuse([bm25_ids, vector_ids]))
400 font-semibold">class=400 font-semibold">class="text-emerald-300">"text-slate-500 italic"># 330 and 101 rise to the top because both retrievers rank them well

Worked example: document 330 is rank 3 in BM25 and rank 1 in vector search, so its score is 1/63 + 1/61, about 0.0323. Document 777, present only in the vector list at rank 2, scores about 0.0161. Agreement beats a single strong vote.

Two tuning levers are worth knowing:

• Weights. For identifier-heavy catalogs (parts, electronics), weight BM25 at 1.0 and vector at 0.6. For fashion and home goods, invert it.
• Query-aware routing. If a query matches a SKU pattern or a known brand-plus-model, boost the lexical list or skip vector retrieval altogether. A tiny classifier or a regex set pays for itself.

OpenSearch also offers a native hybrid query with search pipelines (a normalization processor, and in recent releases rank-based fusion), which keeps fusion inside the cluster. Fusing in your application layer is equally valid and easier to test and version.

7. Reranking: Spend Model Compute Only Where It Pays#

After fusion you have a good top 50. A cross-encoder reranker, or a learning-to-rank model using behavioural features (click-through, conversion, margin, freshness), can reorder it. Rules of thumb:

• Rerank 50 candidates at most; latency grows linearly with candidates.
• Set a hard timeout (for example 80 ms) and fall back to the fused order on breach. Search that degrades gracefully beats search that is occasionally perfect and occasionally down.

8. Going Global Without Going Slow#

Global means three separate problems: geography, language and inventory.

• Geography. Deploy read replicas or search clusters in each major region and route by latency. Replicate the index asynchronously, and replicate stock and price on a faster path than the full document so the catalog stays honest.
• Language. Use a multilingual embedding model so a query in Spanish can match an English-titled product, and keep per-language analyzers for the lexical side. Evaluate each language separately; averages hide weak locales.
• Inventory and locale filters. Region, currency and shippability are hard filters, never ranking hints. Putting them in the filter set (Section 5) is what makes the results trustworthy.
• Caching. Cache normalised query embeddings (head queries repeat heavily) and, for the head of the query distribution, the fused result IDs for 30 to 120 seconds. In many catalogs a minority of distinct queries accounts for a majority of traffic, so even modest caching removes the embedding stage for most requests.

Rough sizing for planning: a 1024-dimension float32 vector is about 4 KB, so 10 million products need roughly 40 GB for raw vectors before graph overhead. Scalar or product quantization, or lower-dimension models, can cut that by a factor of four or more at a small recall cost that you should measure on your own data.

9. Evaluation: Prove It Before You Ship It#

Do not tune by anecdote. Build an evaluation harness before the first production cutover:

1. Judgement set. 500 to 2,000 real queries stratified by type (head, torso, long tail, identifier, zero-result), each with graded relevance labels from merchandisers or a reviewed LLM-assisted process.
2. Offline metrics. NDCG@10, recall@100 for the retrievers, zero-result rate and mean reciprocal rank.
3. Online metrics. Search-to-cart rate, null-search rate, reformulation rate and revenue per search session, via A/B test with a holdback group.
4. Latency as a metric. Track p50, p95 and p99 per stage, not only end-to-end.

10. Failure Modes to Plan For#

• Embedding service slow or down: apply a timeout and fall back to BM25-only results.
• Over-selective filter on ANN: use engine-native filtering, iterative scans or an exact-search fallback.
• Stale vectors after a template change: version embeddings and dual-run during migration.
• Identifier queries missing: route SKU-like queries to the lexical retriever.
• Cache poisoning across locales: include locale and filters in the cache key.

11. Decision Guide#

Conclusion: Treat Search as a Product, Not a Plugin#

Hybrid search is not a model choice; it is a system design. The teams that get durable results do five things consistently: they run lexical and semantic retrieval in parallel under identical filters, they fuse with rank-based methods like RRF, they treat embedding and reranking as latency-budgeted dependencies with fallbacks, they measure recall and relevance against a real judgement set, and they route queries instead of assuming one retriever fits all.

If you are weighing a pgvector build against OpenSearch, or need to retrofit semantic discovery onto an existing catalog without destabilising conversion, knetwork's engineering team can help with architecture reviews, evaluation harness design and phased implementation. A short consultation is usually enough to size the effort and de-risk the first release.