A retrieval router that classifies every query and dispatches it to sparse BM25, dense FAISS, an RRF-fused hybrid, or live web search — instead of paying always-dense or always-hybrid cost on every request. Every decision is logged, benchmarked against public IR datasets, and tunable to your corpus.
# classify a query without spending a token
$ curl -s 'https://arr.dev/router/explain?query=BM25+rank+fusion'
{
"query": "BM25 rank fusion",
"route": "sparse",
"confidence": 0.6,
"profile": "balanced",
"matched_rule": {
"id": 3, "name": "short_keyword",
"condition": "<= 3 tokens, no stopwords removed"
},
"tokens": ["bm25", "rank", "fusion"],
"latency_ms": 0.4,
"cost_usd": 0.0
} This query would have paid a dense embedding call under always-hybrid. Here it costs $0.00 and 0.4 ms.
Each backend is strong somewhere and wasteful somewhere else. Pick one for the whole corpus and you either overpay or under-answer. The evidence, backend by backend:
So route per query. Classify first, then dispatch to the cheapest backend likely to answer well — and log every decision so the routing itself is auditable.
The default router is a heuristic classifier — deliberately simple and fully inspectable. Watch a query flow through it, or read the rule table it runs.
| # | condition | route | conf |
|---|---|---|---|
| 1 | Freshness keyword (latest / current / today / this week / right now) or year ≥ current | web | 0.9 |
| 2 | keyword_overlap_ratio > θ and short token count | sparse | 0.7 |
| 3 | ≤ 3 tokens, no stopwords removed | sparse | 0.6 |
| 4 | > 12 tokens, or contains how / why / what / explain | dense | 0.7 |
| 5 | otherwise | hybrid | 0.5 |
Tokenization is deliberately simple: lowercase → split on non-alphanumeric, minus a hand-curated 30-word stopword set. Nothing learned, nothing opaque — every decision is reproducible from the query string alone, at GET /router/explain, without spending a token.
Same interface, four cost/latency profiles. The router's whole job is to spend the expensive ones only when they'll earn it.
| backend | engine | latency | cost / query | best for |
|---|---|---|---|---|
| sparse | rank_bm25 · BM25Okapi | 1–5 ms | $0 | keyword, named-entity, exact term |
| dense | FAISS IndexFlatIP + text-embedding-3-small | 50–200 ms | ~$0.0000001 | semantic, paraphrase |
| hybrid | dense + sparse, RRF (k=60), run in parallel | 50–200 ms | dense cost only | balanced default |
| web | Tavily basic search | 200–600 ms | $0.008 flat | real-time / out-of-corpus |
Two public BEIR datasets with human qrels. The question isn't "is adaptive good" — it's exactly how much quality a per-query decision buys back the latency and embedding cost it saves.
323 queries · 3,633 docs · lower-left is cheaper & faster, upper is more relevant.
| strategy | nDCG@10 | Recall@10 | latency | embed/q |
|---|---|---|---|---|
| always-sparse | 0.306 | 0.152 | 8.0 ms | 0.00 |
| always-dense | 0.384 | 0.184 | 202.0 ms | 1.00 |
| always-hybrid | 0.375 | 0.180 | 190.0 ms | 1.00 |
| adaptive · speed | 0.313 | 0.155 | 20.8 ms | 0.07 |
| adaptive · balanced | 0.328 | 0.160 | 44.5 ms | 0.20 |
| adaptive · quality | 0.377 | 0.181 | 189.9 ms | 0.96 |
Reading: balanced sends ~80% of short keyword queries to free 8 ms BM25 — 4.3× faster than always-hybrid with 80% fewer embedding calls — at a cost of 0.056 nDCG versus always-dense. quality closes that gap to 0.007 nDCG of always-dense while beating always-hybrid outright.
On 300 longer claim-verification queries, the per-query choice doesn't just trade quality for speed — it wins outright. Adaptive balanced scores a higher nDCG@10 than every fixed backend, including always-dense and always-hybrid. When the workload is mixed, choosing per query beats committing to any single strategy.
This is the case for adaptive routing in one number: 0.743 > 0.742 > 0.731 > 0.652.
| strategy | nDCG@10 | Δ vs balanced |
|---|---|---|
| adaptive · balanced | 0.743 | — |
| always-hybrid | 0.742 | −0.001 |
| always-dense | 0.731 | −0.012 |
| always-sparse | 0.652 | −0.091 |
Thresholds are a selectable RouterParameters bundle, not magic constants — validated by a 208-config sweep across NFCorpus + SciFact, and stamped on every decision.
Max free BM25 traffic, lowest latency. Sends the most queries to $0 sparse.
The original shipped behavior. Cheap where it's safe, dense where it counts.
Only near-certain keyword hits bypass embeddings; everything else goes hybrid or dense.
calibrate_router.py runs retrieval once per backend, then scores hundreds of candidate configs by pure re-routing over the cached results — so the whole sweep costs no more than a single benchmark pass. It emits the winning config plus ready-to-paste env lines.
Ground-truth sources: BEIR qrels · an existing judge parquet (zero API cost) · or a public BEIR dataset.
$ python scripts/calibrate_router.py \
--qrels data/mycorpus/qrels.tsv \
--profile-out .env.router
sweeping 208 configs over cached retrieval…
best nDCG@10 0.741 (Δ +0.018 vs balanced)
wrote .env.router — paste to deploy:
ROUTER_THETA=0.42
ROUTER_SHORT_TOKENS=4
ROUTER_PROFILE=customThe heuristic router is the front door. Behind it, each of these is a real module built for production retrieval.
Runs free ~10 ms sparse first, inspects the score profile — the top-1 floor and the relative margin between the top and k-th hit — and only pays for dense embedding + RRF when lexical evidence is weak. It decides on measured results, not query shape, so it doesn't degrade as corpus vocabulary grows.
FAISS index + doc map keyed by SHA-256(doc_id, content). Corpus edits auto-invalidate. First boot embeds (~$0.0003); later boots load from disk in milliseconds.
On the hot path: long TTL for embeddings (saves an OpenAI round trip), short TTL for web results (freshness). ttl=0 disables cleanly via env.
Sync retrieve across all four backends, then async LLM-judge on a 1/2/3 rubric — checkpointed every 50 queries, so a mid-run crash loses nothing.
Returns inline [n] citations. stream=true emits SSE — meta → delta → done — with token usage and cost in the final event.
One JSON line per /retrieve: query_hash (sha1, first 8 hex — correlate without leaking query text), backend, latency_ms, cost_usd.
Everything above ships open. When you need more, it's already wired:
A small, honest HTTP surface. FastAPI with frozen Pydantic v2 models — the same contract the product UI and the ops dashboard call.
stream=true emits meta → delta → done.$ curl -s https://arr.dev/retrieve \
-H 'content-type: application/json' \
-d '{"query": "cochrane review vitamin d", "k": 5}'
{
"backend": "sparse", # router chose free BM25
"profile": "balanced",
"latency_ms": 3.1,
"cost_usd": 0.0,
"hits": [
{ "doc_id": "MED-2419", "score": 18.7 },
{ "doc_id": "MED-0088", "score": 15.2 }
]
}Boring, legible infrastructure. Every layer is something you'd choose yourself.