simlar on BEIR: Hybrid Retrieval Across 14 Datasets

Every retrieval engine claims to be fast and accurate. The only way to know what that costs you is to run the same 14 BEIR datasets through it — from 3,633-document nfcorpus up to 8.8-million-document msmarco — and look at the numbers dataset by dataset, not just the average. This benchmark asks whether simlar's hybrid retrieval is actually worth it against plain dense search: brute-force exact search, FAISS, and turbovec.

Methodology

All four engines were evaluated against the same 14 BEIR datasets, using the same embedding model, WhereIsAI/UAE-Large-V1, and top_k=100. We report five standard BEIR metrics, averaged unweighted across datasets for the headline numbers: NDCG@100 (ranking quality within the top 100 results), Recall@100 (share of relevant documents retrieved in the top 100), P@100 (precision at 100), MAP, and MRR. Latency is retrieval_seconds, the total wall-clock time to answer every query in a dataset, which we also divide by query count for a per-query figure.

There's one important difference in how the four are configured. Brute force, FAISS, and turbovec run dense vector retrieval only, while simlar uses its native hybrid pipeline, combining text and vector retrieval with a text_k/vector_k split configured per dataset. The results reflect each engine's own configured retrieval path, not simlar forced into a dense-only mode to make the comparison symmetric.

turbovec didn't complete on msmarco — the process was killed before it produced results, almost certainly memory pressure on an 8.8-million-document corpus. So turbovec has results for 13 of the 14 datasets, and its macro-averages below are computed over those 13. Total retrieval-time comparisons exclude msmarco for all four engines, so every engine is measured over the same 13-dataset subset there.

simlar vs. brute force, FAISS, and turbovec

The headline result isn't really about accuracy — on ranking quality the four are close. It's about what happens to query latency as the corpus grows from thousands of documents to millions.

† turbovec's row is a 13-dataset average (missing msmarco) — not directly comparable to the other three rows' 14-dataset average.

Reading the numbers

1. simlar is the only one of the four that gets relatively faster as the corpus grows, not slower. On the two largest corpora — fever (5.4M docs) and hotpotqa (5.2M docs) — simlar answers both in 42.6 seconds combined. FAISS takes 915.9 seconds for the same two; brute-force exact search takes 419.4 seconds. Summed across all 14 datasets, simlar's total retrieval time is 62.6s against 564.6s for brute force and 1,358.6s for FAISS.
2. Brute-force and FAISS land on nearly identical accuracy, because in this configuration they're doing the same math. Their macro-average NDCG@100 differs by 0.0001 (0.5602 vs. 0.5601), and per-dataset numbers match to three decimal places almost everywhere — e.g. hotpotqa 0.7626 vs. 0.7626, fever 0.8416 vs. 0.8416. FAISS here runs an exact index, so any gap between the two is pure implementation overhead, not a search-quality difference — and FAISS is consistently the slower of the pair despite computing an identical ranking (dbpedia-entity: 13.8s brute force vs. 105.4s FAISS).
3. simlar wins Recall@100 and MRR, ties NDCG@100, and gives up a little MAP. Pruning the candidate set down to a smaller text_k/vector_k instead of scanning everything costs about a point of MAP (0.399 vs. 0.407–0.408) in exchange for the latency win above — an honest trade, not a free one.
4. turbovec is the only real competitor on speed, and simlar still leads it by 1.9–2.9× on the largest datasets. turbovec's 4-bit quantized vectors get it to 5.95ms/query on hotpotqa and 6.47ms on climate-fever; simlar answers the same queries in 3.19ms and 3.43ms. Where turbovec pulls ahead is small corpora with sparse-but-precise matches — arguana (8,674 docs), where turbovec's 0.04ms beats simlar's 0.20ms.
5. The single biggest swing in the whole table is trec-covid. simlar's MRR there is 0.955 against 0.801–0.811 for the three dense-only baselines. With only 50 queries and a corpus of dense biomedical literature, the keyword half of simlar's hybrid fusion is doing real work that none of the other three have access to.

Picking a lane

simlar's hybrid retrieval is the only one of the four whose query latency stays flat as the corpus scales into the millions, and it does that while matching or beating the three dense-only baselines on Recall@100 and MRR — the honest cost is a small amount of MAP. turbovec is the closest thing to a real competitor on speed, but simlar still leads it by 1.9–2.9× on the largest datasets. Brute-force and FAISS are worth keeping around as a ground truth — they're not competing on latency, they're confirming everyone else's ranking is correct.