The Problem
Everyone ships search; almost nobody measures it. Teams bolt vectors onto an index, eyeball a few queries, and call it "semantic search" — with no way to know whether relevance got better, worse, or just different. The questions that actually matter are empirical: does hybrid retrieval beat lexical alone? Does a reranker earn its latency? Is Elasticsearch worth running when Postgres is already on the stack?
Every retrieval opinion here had to survive contact with hand-labeled relevance judgments.
The Approach
SearchLab is a relevance-engineering harness built around one discipline: every change is a hypothesis scored offline against a public benchmark (BEIR SciFact — 5k scientific abstracts, 300 queries, expert qrels) using standard IR metrics from ir_measures, so the numbers stay comparable to published baselines.
- Backend-agnostic by construction. Retrieval backends implement a small
Retrieverprotocol; the eval harness, qrels, and metrics are identical across them. Swapping Elasticsearch for Postgres changes one flag, not the experiment. - Two arms, then fusion. A BM25 arm over an inverted index and a dense arm (MiniLM sentence embeddings under HNSW), fused with reciprocal-rank fusion — license-free, parameter-light, and bit-identical to Elastic's native RRF.
- A second stage on trial. A cross-encoder reranker rescores the candidate pool — treated as an experiment, not a default.
- Latency beside relevance. Every run reports client-side p50/p95 alongside nDCG, so quality wins carry their cost.
The Scoreboard
| Backend | Mode | nDCG@10 | Recall@100 | MRR | p50 ms |
|---|---|---|---|---|---|
| Elastic | BM25 | 0.6537 | 0.8846 | 0.6236 | 21 |
| Elastic | BM25 + rerank | 0.6743 | 0.8846 | 0.6539 | * |
| Elastic | vector | 0.6484 | 0.9250 | 0.6123 | 34 |
| Elastic | hybrid (RRF) | 0.6896 | 0.9543 | 0.6612 | 58 |
| Elastic | hybrid + rerank | 0.6862 | 0.9543 | 0.6611 | * |
| Postgres | FTS | 0.2975 | 0.7459 | 0.2603 | 21 |
| Postgres | vector | 0.6484 | 0.9250 | 0.6123 | 31 |
| Postgres | hybrid (RRF) | 0.5763 | 0.9260 | 0.5487 | 62 |
* Rerank p50 is ≈ 2–3 s on CPU — two orders of magnitude above the retrieval rows and off-scale for this column. The cross-encoder cost is measured and discussed in the notebook's reranking section.
What the Numbers Said
- The baseline validates the harness. BM25 at 0.6537 nDCG@10 lands on the published BEIR figure — the entire pipeline is trustworthy before any experiment runs on it.
- Hybrid wins because the arms are complementary. Fusion beats both parents (+3.6 nDCG points, +7 recall points over BM25) — lexical and semantic retrieval fail on different queries.
- Reranking is conditional, not free lift. It rescues a weak lexical run (+2.1 points) and slightly hurts an already-strong hybrid (−0.3) — a result you only catch by measuring.
- Elasticsearch's edge is BM25 quality, not speed. With identical vectors, the dense arms tie to four decimals and latency is comparable; the entire gap is Postgres FTS (0.2975) vs real BM25 (0.6537) — which is also why the weak arm poisons Postgres's hybrid.
The Outcome
A defensible recommendation — hybrid RRF on Elasticsearch, reranking only where a per-workload experiment earns it — plus a harness where the next question (a better encoder, a learned-sparse arm, another dataset) is one CLI flag and one afternoon away. The notebook walks the per-query evidence behind every claim above.