The following is the full notebook behind our RAG retrieval benchmark — every markdown cell and every line of code, in the order it ran, along with the actual output.
All four notebooks are public, so you can run them yourself. This post walks through 10_rags_benchmark.ipynb, which benchmarks sim_LAR through all three framework integrations at once — but each integration also has its own dedicated notebook:
- 07_langchain_benchmark.ipynb — LangChain integration.
- 08_llamaindex_benchmark.ipynb — LlamaIndex integration.
- 09_haystack_benchmark.ipynb — Haystack integration.
- 10_rags_benchmark.ipynb — the combined benchmark covered in this post.
10 — RAG Retriever Benchmark
This notebook evaluates simlar against other vector stores in a retrieval-augmented generation (RAG) setting. We measure how well each retriever finds the correct document for a given question — the core task in RAG before any LLM is involved.
What we cover
- Loading a question-answering dataset and building a ground-truth corpus
- Evaluating retrieval quality with Hit@1, Hit@k, and MRR
- Comparing simlar against Chroma, Qdrant, and Milvus
- Measuring index build time and query latency
%pip install -q \
sentence-transformers==5.6.0 datasets==5.0.0 \
langchain-core==1.4.8 langchain-huggingface==1.2.2 langchain-chroma==1.1.0 langchain-community==0.4.2 langchain-deepseek==1.1.0 \
qdrant-client==1.18.0 \
pymilvus==3.0.0 pymilvus[milvus_lite]==3.0.0 \
pinecone==9.1.0 \
llama-index-core==0.14.23 llama-index-embeddings-huggingface==0.7.0 llama-index-llms-openai-like==0.7.2 \
haystack-ai==2.31.0 numba==0.65.1 faiss-cpu==1.14.3 turbovec==0.8.0Environment variables
API keys for Pinecone and DeepSeek. Each prompt only appears if the key is not already set in the environment.
import os
from getpass import getpass
if not os.environ.get("PINECONE_API_KEY"):
os.environ["PINECONE_API_KEY"] = getpass("Pinecone API key: ")
if not os.environ.get("DEEPSEEK_API_KEY"):
os.environ["DEEPSEEK_API_KEY"] = getpass("DeepSeek API key: ")Dataset
We use Kaggle — a reading comprehension dataset where each question has a corresponding context passage that contains the answer. This structure maps naturally to RAG: the context passages become the corpus and the questions become the queries. For each query we know exactly which document should be retrieved, giving us a ground truth to evaluate against. Place the CSV file in the same directory as this notebook before running.
from datasets import load_dataset
from abc import ABC, abstractmethod
class Dataset(ABC):
corpus: list[str]
queries: list[str]
@abstractmethod
def evaluate(self, query: str, docs: list[str]) -> bool: ...
@abstractmethod
def rank(self, query: str, docs: list[str]) -> int | None: ...
class SquadDataset(Dataset):
def __init__(self, corpus, queries, query_to_context):
self.corpus = corpus
self.queries = queries
self._query_to_context = query_to_context
@classmethod
def from_hf(cls, n_questions=None, n_contexts=None):
hf = load_dataset("rajpurkar/squad", split="train")
seen, corpus, queries, query_to_context = set(), [], [], {}
for row in hf:
if n_questions is not None and len(queries) == n_questions:
break
ctx = row["context"]
if ctx not in seen:
if n_contexts is not None and len(corpus) >= n_contexts:
continue
seen.add(ctx)
corpus.append(ctx)
queries.append(row["question"])
query_to_context[row["question"]] = ctx
return cls(corpus=corpus, queries=queries, query_to_context=query_to_context)
def evaluate(self, query, docs):
return self._query_to_context.get(query) in docs
def rank(self, query: str, docs: list[str]) -> int | None:
correct = self._query_to_context.get(query)
for i, doc in enumerate(docs):
if doc == correct:
return i
return None
class MsMarcoDataset(Dataset):
def __init__(self, corpus, queries, query_to_selected):
self.corpus = corpus
self.queries = queries
self._query_to_selected = query_to_selected
@classmethod
def from_hf(cls, n_queries=None, n_passages=None):
hf = load_dataset("microsoft/ms_marco", "v2.1", split="train")
seen, passages, queries, query_to_selected = set(), [], [], {}
for row in hf:
if n_passages is None or len(passages) < n_passages:
for text in row["passages"]["passage_text"]:
if text not in seen:
seen.add(text)
passages.append(text)
if n_queries is None or len(queries) < n_queries:
selected = {t for t, s in zip(row["passages"]["passage_text"], row["passages"]["is_selected"]) if s == 1}
if selected and selected & set(passages):
queries.append(row["query"])
query_to_selected[row["query"]] = selected & set(passages)
if (n_passages is None or len(passages) >= n_passages) and \
(n_queries is None or len(queries) >= n_queries):
break
return cls(corpus=passages, queries=queries, query_to_selected=query_to_selected)
def evaluate(self, query, docs):
return bool(self._query_to_selected.get(query, set()) & set(docs))
def rank(self, query, docs):
selected = self._query_to_selected.get(query, set())
for i, doc in enumerate(docs):
if doc in selected:
return i
return NoneIndexing Techniques
Each technique wraps a vector store and exposes a uniform interface: given a query string, return the top-k most relevant documents from the corpus. We test simlar across its three integration layers — LangChain, LlamaIndex, and Haystack — and compare against Chroma, Qdrant, and Milvus.
All suits use the same embedding model (all-MiniLM-L6-v2) so that differences in retrieval quality reflect the underlying index, not the embeddings.
from typing import Protocol
class HaystackBaseRetriever(Protocol):
def run(self, query: str) -> dict: ...simlar
simlar exposes a hybrid index that combines keyword and semantic (vector) search. It integrates natively with LangChain, LlamaIndex, and Haystack — the three classes below each wrap the same simlar index through a different integration layer.
from langchain_core.documents import Document
from langchain_core.retrievers import BaseRetriever as LangchainBaseRetriever
from llama_index.core.retrievers import BaseRetriever as LlamaBaseRetriever
from llama_index.core.schema import NodeWithScore, TextNode, QueryBundle
class SimlarTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from langchain_huggingface import HuggingFaceEmbeddings
from simlar.integrations.langchain.simlar_vector_store import SimlarVectorStore
from simlar.integrations.langchain.langchain_retriever import SimlarRetriever
SimlarRetriever.model_rebuild()
store = SimlarVectorStore.from_texts(
texts=dataset.corpus,
embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
)
return SimlarRetriever(vector_store=store, k=k)
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
import numpy as np
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from simlar.integrations.llama_index.simlar_retriever import SimlarRetriever
embed_model = HuggingFaceEmbedding(model_name="all-MiniLM-L6-v2")
vectors = np.array(embed_model.get_text_embedding_batch(dataset.corpus), dtype=np.float32)
return SimlarRetriever.from_texts(
texts=dataset.corpus,
ids=[str(i) for i in range(len(dataset.corpus))],
vectors=vectors,
embed_model=embed_model,
k=k,
)
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
from haystack import Document as HaystackDocument, component
from haystack.components.embedders import (
SentenceTransformersDocumentEmbedder,
SentenceTransformersTextEmbedder,
)
from simlar.integrations.haystack.simlar_document_store import SimlarDocumentStore
from simlar.integrations.haystack.simlar_retriever import SimlarHybridRetriever
store = SimlarDocumentStore(top_k=k)
doc_embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2", progress_bar=False)
doc_embedder.warm_up()
store.write_documents(
doc_embedder.run([HaystackDocument(content=t) for t in dataset.corpus])["documents"]
)
_retriever = SimlarHybridRetriever(document_store=store, top_k=k)
_text_embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2", progress_bar=False)
_text_embedder.warm_up()
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
embedding = _text_embedder.run(query)["embedding"]
return _retriever.run(query=query, query_embedding=embedding)
return _R()Chroma
Chroma is an open-source embedding database designed for AI applications. It stores vectors in memory or on disk and supports similarity search out of the box. In this benchmark we use it in in-memory mode so there is no persistence overhead between runs.
class ChromaTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings
store = Chroma.from_texts(
texts=dataset.corpus,
embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
)
return store.as_retriever(search_kwargs={"k": k})
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings
_store = Chroma.from_texts(
texts=dataset.corpus,
embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
)
class _R(LlamaBaseRetriever):
def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
return [NodeWithScore(node=TextNode(text=d.page_content))
for d in _store.similarity_search(query_bundle.query_str, k=k)]
return _R()
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
from haystack import Document as HaystackDocument, component
from langchain_chroma import Chroma
from langchain_huggingface import HuggingFaceEmbeddings
_store = Chroma.from_texts(
texts=dataset.corpus,
embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
)
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
return {"documents": [HaystackDocument(content=d.page_content)
for d in _store.similarity_search(query, k=k)]}
return _R()Qdrant
Qdrant is a vector search engine built in Rust, optimized for high-performance similarity search. It supports filtering, payloads, and multiple distance metrics. Here we run it in in-memory mode (":memory:") to keep setup simple and avoid disk I/O in the benchmark.
class QdrantTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
_client = QdrantClient(":memory:")
_client.create_collection(
collection_name="corpus",
vectors_config=VectorParams(size=vectors.shape[1], distance=Distance.COSINE),
)
_client.upsert(
collection_name="corpus",
points=[PointStruct(id=i, vector=vectors[i].tolist(), payload={"text": text})
for i, text in enumerate(_corpus)],
)
class _R(LangchainBaseRetriever):
def _get_relevant_documents(self, query, *, run_manager=None):
qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
return [Document(page_content=r.payload["text"])
for r in _client.query_points(collection_name="corpus", query=qv, limit=k).points]
return _R()
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
_client = QdrantClient(":memory:")
_client.create_collection(
collection_name="corpus",
vectors_config=VectorParams(size=vectors.shape[1], distance=Distance.COSINE),
)
_client.upsert(
collection_name="corpus",
points=[PointStruct(id=i, vector=vectors[i].tolist(), payload={"text": text})
for i, text in enumerate(_corpus)],
)
class _R(LlamaBaseRetriever):
def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
qv = _model.encode([query_bundle.query_str], normalize_embeddings=True)[0].tolist()
results = _client.query_points(collection_name="corpus", query=qv, limit=k).points
return [NodeWithScore(node=TextNode(text=r.payload["text"]), score=r.score) for r in results]
return _R()
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
from haystack import Document as HaystackDocument, component
from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
_client = QdrantClient(":memory:")
_client.create_collection(
collection_name="corpus",
vectors_config=VectorParams(size=vectors.shape[1], distance=Distance.COSINE),
)
_client.upsert(
collection_name="corpus",
points=[PointStruct(id=i, vector=vectors[i].tolist(), payload={"text": text})
for i, text in enumerate(_corpus)],
)
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
results = _client.query_points(collection_name="corpus", query=qv, limit=k).points
return {"documents": [HaystackDocument(content=r.payload["text"]) for r in results]}
return _R()Milvus
Milvus is a distributed vector database built for large-scale similarity search. We use Milvus Lite — a lightweight version that runs locally as a file-based database, with no server required. Note that only one process can open the database file at a time.
class MilvusTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from sentence_transformers import SentenceTransformer
from pymilvus import MilvusClient
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
_client = MilvusClient("/tmp/milvus_rags.db")
if _client.has_collection("corpus"):
_client.drop_collection("corpus")
_client.create_collection(collection_name="corpus", dimension=vectors.shape[1])
_client.insert(
collection_name="corpus",
data=[{"id": i, "vector": vectors[i].tolist(), "text": text}
for i, text in enumerate(_corpus)],
)
class _R(LangchainBaseRetriever):
def _get_relevant_documents(self, query, *, run_manager=None):
qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
results = _client.search(collection_name="corpus", data=[qv], limit=k, output_fields=["text"])
return [Document(page_content=r["entity"]["text"]) for r in results[0]]
return _R()
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
from sentence_transformers import SentenceTransformer
from pymilvus import MilvusClient
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
_client = MilvusClient("/tmp/milvus_rags.db")
if _client.has_collection("corpus"):
_client.drop_collection("corpus")
_client.create_collection(collection_name="corpus", dimension=vectors.shape[1])
_client.insert(
collection_name="corpus",
data=[{"id": i, "vector": vectors[i].tolist(), "text": text}
for i, text in enumerate(_corpus)],
)
class _R(LlamaBaseRetriever):
def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
qv = _model.encode([query_bundle.query_str], normalize_embeddings=True)[0].tolist()
results = _client.search(collection_name="corpus", data=[qv], limit=k, output_fields=["text"])
return [NodeWithScore(node=TextNode(text=r["entity"]["text"])) for r in results[0]]
return _R()
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
from haystack import Document as HaystackDocument, component
from sentence_transformers import SentenceTransformer
from pymilvus import MilvusClient
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
_client = MilvusClient("/tmp/milvus_rags.db")
if _client.has_collection("corpus"):
_client.drop_collection("corpus")
_client.create_collection(collection_name="corpus", dimension=vectors.shape[1])
_client.insert(
collection_name="corpus",
data=[{"id": i, "vector": vectors[i].tolist(), "text": text}
for i, text in enumerate(_corpus)],
)
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
results = _client.search(collection_name="corpus", data=[qv], limit=k, output_fields=["text"])
return {"documents": [HaystackDocument(content=r["entity"]["text"]) for r in results[0]]}
return _R()Pinecone
Pinecone is a managed vector database — unlike the others, it runs as an external cloud service. This means build time includes network latency for uploading vectors, and query time includes a round-trip to Pinecone's servers. A PINECONE_API_KEY environment variable is required.
class PineconeTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from sentence_transformers import SentenceTransformer
from pinecone import Pinecone, ServerlessSpec
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
pc = Pinecone()
if "rag-corpus" not in [i.name for i in pc.list_indexes()]:
pc.create_index(
name="rag-corpus",
dimension=vectors.shape[1],
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
_index = pc.Index("rag-corpus")
batch_size = 100
records = [{"id": str(i), "values": vectors[i].tolist(), "metadata": {"text": text}}
for i, text in enumerate(_corpus)]
for i in range(0, len(records), batch_size):
_index.upsert(vectors=records[i:i + batch_size])
class _R(LangchainBaseRetriever):
def _get_relevant_documents(self, query, *, run_manager=None):
qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
results = _index.query(vector=qv, top_k=k, include_metadata=True)
return [Document(page_content=m["metadata"]["text"]) for m in results["matches"]]
return _R()
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
from sentence_transformers import SentenceTransformer
from pinecone import Pinecone, ServerlessSpec
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
pc = Pinecone()
if "rag-corpus" not in [i.name for i in pc.list_indexes()]:
pc.create_index(
name="rag-corpus",
dimension=vectors.shape[1],
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
_index = pc.Index("rag-corpus")
batch_size = 100
records = [{"id": str(i), "values": vectors[i].tolist(), "metadata": {"text": text}}
for i, text in enumerate(_corpus)]
for i in range(0, len(records), batch_size):
_index.upsert(vectors=records[i:i + batch_size])
class _R(LlamaBaseRetriever):
def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
qv = _model.encode([query_bundle.query_str], normalize_embeddings=True)[0].tolist()
results = _index.query(vector=qv, top_k=k, include_metadata=True)
return [NodeWithScore(node=TextNode(text=m["metadata"]["text"])) for m in results["matches"]]
return _R()
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
from haystack import Document as HaystackDocument, component
from sentence_transformers import SentenceTransformer
from pinecone import Pinecone, ServerlessSpec
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True)
pc = Pinecone()
if "rag-corpus" not in [i.name for i in pc.list_indexes()]:
pc.create_index(
name="rag-corpus",
dimension=vectors.shape[1],
metric="cosine",
spec=ServerlessSpec(cloud="aws", region="us-east-1"),
)
_index = pc.Index("rag-corpus")
batch_size = 100
records = [{"id": str(i), "values": vectors[i].tolist(), "metadata": {"text": text}}
for i, text in enumerate(_corpus)]
for i in range(0, len(records), batch_size):
_index.upsert(vectors=records[i:i + batch_size])
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
results = _index.query(vector=qv, top_k=k, include_metadata=True)
return {"documents": [HaystackDocument(content=m["metadata"]["text"]) for m in results["matches"]]}
return _R()FAISS
FAISS (Facebook AI Similarity Search) is a library for efficient similarity search over dense vectors. It runs entirely in memory with no server required, using optimized CPU/GPU kernels. Unlike the other stores in this benchmark, FAISS is pure vector search — no keyword component — making it a useful baseline for semantic-only retrieval.
class FaissTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from langchain_community.vectorstores import FAISS as FaissStore
from langchain_huggingface import HuggingFaceEmbeddings
store = FaissStore.from_texts(
texts=dataset.corpus,
embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
)
return store.as_retriever(search_kwargs={"k": k})
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
import faiss
import numpy as np
from sentence_transformers import SentenceTransformer
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True).astype(np.float32)
_index = faiss.IndexFlatIP(vectors.shape[1])
_index.add(vectors)
class _R(LlamaBaseRetriever):
def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
qv = _model.encode([query_bundle.query_str], normalize_embeddings=True).astype(np.float32)
scores, indices = _index.search(qv, k)
return [NodeWithScore(node=TextNode(text=_corpus[i]), score=float(scores[0][j]))
for j, i in enumerate(indices[0]) if i >= 0]
return _R()
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
import faiss
import numpy as np
from haystack import Document as HaystackDocument, component
from sentence_transformers import SentenceTransformer
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True).astype(np.float32)
_index = faiss.IndexFlatIP(vectors.shape[1])
_index.add(vectors)
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
qv = _model.encode([query], normalize_embeddings=True).astype(np.float32)
_, indices = _index.search(qv, k)
return {"documents": [HaystackDocument(content=_corpus[i]) for i in indices[0] if i >= 0]}
return _R()turbovec
turbovec is a vector search library implemented in Rust with Python bindings, built on Google Research's TurboQuant algorithm. It uses 2–4 bit quantization to achieve up to 16x compression — a 10M document corpus that takes 31GB as float32 fits in 4GB. Hand-optimized SIMD kernels (AVX-512, NEON) make it faster than FAISS on ARM. Like FAISS, it is pure vector search with no keyword component.
class TurbovecTechnique:
@classmethod
def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
from turbovec.langchain import TurboQuantVectorStore
from langchain_huggingface import HuggingFaceEmbeddings
store = TurboQuantVectorStore.from_texts(
texts=dataset.corpus,
embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
bit_width=4,
)
return store.as_retriever(search_kwargs={"k": k})
@classmethod
def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
import numpy as np
from turbovec import TurboQuantIndex
from sentence_transformers import SentenceTransformer
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True).astype(np.float32)
_index = TurboQuantIndex(dim=vectors.shape[1], bit_width=4)
_index.add(vectors)
class _R(LlamaBaseRetriever):
def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
qv = _model.encode([query_bundle.query_str], normalize_embeddings=True).astype(np.float32)
scores, indices = _index.search(qv, k=k)
return [NodeWithScore(node=TextNode(text=_corpus[i]), score=float(scores[0][j]))
for j, i in enumerate(indices[0]) if i >= 0]
return _R()
@classmethod
def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
import numpy as np
from turbovec import TurboQuantIndex
from haystack import Document as HaystackDocument, component
from sentence_transformers import SentenceTransformer
_model = SentenceTransformer("all-MiniLM-L6-v2")
_corpus = dataset.corpus
vectors = _model.encode(_corpus, normalize_embeddings=True).astype(np.float32)
_index = TurboQuantIndex(dim=vectors.shape[1], bit_width=4)
_index.add(vectors)
@component
class _R:
@component.output_types(documents=list[HaystackDocument])
def run(self, query: str) -> dict:
qv = _model.encode([query], normalize_embeddings=True).astype(np.float32)
_, indices = _index.search(qv, k=k)
return {"documents": [HaystackDocument(content=_corpus[i]) for i in indices[0] if i >= 0]}
return _R()RAG
Each RAG class wraps a retriever and exposes two methods:
retrieve(query)— returns the top-k documents as strings, no LLM involved. Used for the benchmark.ask(query)— runs the full RAG pipeline with an LLM and returns an answer.
from dataclasses import dataclass
@dataclass
class AskResult:
question: str
answer: str | None
docs: list[str]
hit: bool
class RAG:
def retrieve(self, query: str) -> list[str]: ...
def ask(self, query: str) -> AskResult: ...
def batch_retrieve() -> list[list[str]]: ...LangChain RAG
Wraps a LangChain retriever. retrieve() calls the retriever directly. ask() runs the full chain with an LLM.
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_deepseek import ChatDeepSeek
class LangchainRAG(RAG):
def __init__(self, dataset: Dataset, retriever: LangchainBaseRetriever):
self._dataset = dataset
self._retriever = retriever
prompt = ChatPromptTemplate.from_template(
"Answer using only the context below. Be concise.\n\n"
"Context:\n{context}\n\nQuestion: {question}"
)
self._chain = (
RunnableParallel(docs=retriever, question=RunnablePassthrough())
| RunnablePassthrough.assign(context=lambda x: "\n\n".join(d.page_content for d in x["docs"]))
| RunnableParallel(
answer=prompt | ChatDeepSeek(model="deepseek-chat") | StrOutputParser(),
docs=lambda x: x["docs"],
)
)
def retrieve(self, query: str) -> list[str]:
return [d.page_content for d in self._retriever.invoke(query)]
def ask(self, query: str) -> AskResult:
result = self._chain.invoke(query)
docs = [d.page_content for d in result["docs"]]
return AskResult(question=query, answer=result["answer"], docs=docs,
hit=self._dataset.evaluate(query, docs))
def batch_retrieve(self, queries: list[str]) -> list[list[str]]:
results = self._retriever.batch(queries)
return [[d.page_content for d in docs] for docs in results]LlamaIndex RAG
Wraps a LlamaIndex retriever. retrieve() calls the retriever directly. ask() runs the full query engine with an LLM.
from llama_index.core.retrievers import BaseRetriever as LlamaBaseRetriever
class LlamaIndexRAG(RAG):
def __init__(self, dataset: Dataset, retriever: LlamaBaseRetriever):
self._dataset = dataset
self._retriever = retriever
from llama_index.core import Settings
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.openai_like import OpenAILike
Settings.embed_model = HuggingFaceEmbedding(model_name="all-MiniLM-L6-v2")
Settings.llm = OpenAILike(
model="deepseek-chat",
api_base="https://api.deepseek.com/v1",
api_key=__import__("os").environ["DEEPSEEK_API_KEY"],
context_window=64000,
is_chat_model=True,
)
self._query_engine = RetrieverQueryEngine.from_args(retriever=retriever)
def retrieve(self, query: str) -> list[str]:
return [n.node.text for n in self._retriever.retrieve(query)]
def ask(self, query: str) -> AskResult:
response = self._query_engine.query(query)
docs = [n.node.text for n in response.source_nodes]
return AskResult(question=query, answer=str(response), docs=docs,
hit=self._dataset.evaluate(query, docs))Haystack RAG
Wraps a Haystack retriever. retrieve() calls the retriever directly. ask() runs a Haystack pipeline with an LLM.
class HaystackRAG(RAG):
def __init__(self, dataset: Dataset, retriever):
self._dataset = dataset
self._retriever = retriever
from haystack import Pipeline
from haystack.components.builders import PromptBuilder
from haystack.components.generators.chat import OpenAIChatGenerator
from haystack.utils import Secret
self._pipeline = Pipeline()
self._pipeline.add_component("prompt", PromptBuilder(
template="Answer using only the context below. Be concise.\n\nContext:\n{% for d in documents %}{{ d.content }}\n{% endfor %}\nQuestion: {{ query }}",
required_variables=["documents", "query"],
))
self._pipeline.add_component("generator", OpenAIChatGenerator(
model="deepseek-chat",
api_base_url="https://api.deepseek.com",
api_key=Secret.from_env_var("DEEPSEEK_API_KEY"),
))
self._pipeline.connect("prompt.prompt", "generator.messages")
def retrieve(self, query: str) -> list[str]:
return [d.content for d in self._retriever.run(query)["documents"]]
def ask(self, query: str) -> AskResult:
docs = self._retriever.run(query)["documents"]
result = self._pipeline.run({"prompt": {"documents": docs, "query": query}})
answer = result["generator"]["replies"][0].text
doc_texts = [d.content for d in docs]
return AskResult(question=query, answer=answer, docs=doc_texts,
hit=self._dataset.evaluate(query, doc_texts))Benchmark
test_retrievers runs the full benchmark in two phases:
1. Build — each retriever indexes the corpus and the elapsed time is recorded.
2. Query — every question in the dataset is passed to rag.retrieve() and the result is evaluated against the ground truth.
Metrics
- Hit@1 — fraction of queries where the correct document is ranked first.
- Hit@k — fraction of queries where the correct document appears in the top k.
- MRR (Mean Reciprocal Rank) — average of
1/rankfor the correct document. Penalizes results that are correct but ranked lower.
import io
import time
import contextlib
import pandas as pd
from typing import Callable
@contextlib.contextmanager
def _suppress():
devnull = os.open(os.devnull, os.O_WRONLY)
old_stdout = os.dup(1)
old_stderr = os.dup(2)
os.dup2(devnull, 1)
os.dup2(devnull, 2)
os.close(devnull)
try:
with contextlib.redirect_stdout(io.StringIO()):
with contextlib.redirect_stderr(io.StringIO()):
yield
finally:
os.dup2(old_stdout, 1)
os.dup2(old_stderr, 2)
os.close(old_stdout)
os.close(old_stderr)
def test_retrievers(
dataset: Dataset,
entries:list[tuple[str, Callable[[], object], Callable[[object], RAG]]],
out:str,
k: int = 3
):
# Build phase
print(f"Dataset: {len(dataset.corpus)} documents & {len(dataset.queries)} queries & {k} k")
rags: list[dict] = []
for name, retriever_factory, rag_factory in entries:
print(f"Building {name}...")
try:
with _suppress():
t0 = time.perf_counter()
retriever = retriever_factory()
build_ms = (time.perf_counter() - t0) * 1000
rag = rag_factory(retriever)
rags.append({"name": name, "rag": rag, "build_ms": build_ms})
except Exception as e:
print(f" skipped: {e}")
# Query phase
n = len(dataset.queries)
for entry in rags:
name, rag = entry["name"], entry["rag"]
hit1 = hitk = mrr = 0.0
errors = 0
t0 = time.perf_counter()
if getattr(rag._retriever, "supports_batch", False):
try:
with _suppress():
all_docs = rag.batch_retrieve(dataset.queries)
for i, (q, docs) in enumerate(zip(dataset.queries, all_docs), 1):
r = dataset.rank(q, docs)
if r is not None:
if r == 0: hit1 += 1
hitk += 1
mrr += 1 / (r + 1)
if i % 100 == 0 or i == n:
print(f"\r {name}: {i}/{n} queries", end="", flush=True)
except Exception as e:
errors += 1
with open("errors.txt", "a") as f:
f.write(f"[{name}] batch error={e}\n")
else:
for i, q in enumerate(dataset.queries, 1):
try:
with _suppress():
docs = rag.retrieve(q)
r = dataset.rank(q, docs)
if r is not None:
if r == 0: hit1 += 1
hitk += 1
mrr += 1 / (r + 1)
except Exception as e:
errors += 1
with open("errors.txt", "a") as f:
f.write(f"[{name}] query={repr(q)} error={e}\n")
if i % 100 == 0 or i == n:
print(f"\r {name}: {i}/{n} queries{f' ({errors} errors)' if errors else ''}", end="", flush=True)
print()
entry["query_ms"] = (time.perf_counter() - t0) * 1000 / n
entry["hit1"] = hit1 / n
entry["hitk"] = hitk / n
entry["mrr"] = mrr / n
# Results
hitk_col = f"Hit@{k}"
rows = [
{
"Retriever": e["name"],
"Hit@1": e["hit1"],
hitk_col: e["hitk"],
"MRR": e["mrr"],
"Build(ms)": e["build_ms"],
"Query(ms)": e["query_ms"],
}
for e in rags
]
df = pd.DataFrame(rows).set_index("Retriever")
for suffix, label in [("langchain", "LangChain"), ("llamaindex", "LlamaIndex"), ("haystack", "Haystack")]:
sub = df[df.index.str.endswith(f"-{suffix}")].copy()
if sub.empty: continue
sub.index = sub.index.str.replace(f"-{suffix}", "", regex=False)
print(f"\n### {label}")
display(sub.style.format({
"Hit@1": "{:.1%}",
hitk_col: "{:.1%}",
"MRR": "{:.3f}",
"Build(ms)": "{:.0f}",
"Query(ms)": "{:.1f}",
}))
# Save to CSV
df.reset_index().rename(columns={
"Retriever": "name", "Hit@1": "hit1", hitk_col: "hitk",
"MRR": "mrr", "Build(ms)": "build_ms", "Query(ms)": "query_ms",
}).to_csv(out, index=False)
print(f"\nSaved to {out}")dataset = MsMarcoDataset.from_hf(n_passages=500_000, n_queries=20_000)
k = 10
entries = [
("simlar-langchain", lambda: SimlarTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("simlar-llamaindex", lambda: SimlarTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
("simlar-haystack", lambda: SimlarTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
("chroma-langchain", lambda: ChromaTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("chroma-llamaindex", lambda: ChromaTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
("chroma-haystack", lambda: ChromaTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
("qdrant-langchain", lambda: QdrantTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("qdrant-llamaindex", lambda: QdrantTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
("qdrant-haystack", lambda: QdrantTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
("milvus-langchain", lambda: MilvusTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("milvus-llamaindex", lambda: MilvusTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
("milvus-haystack", lambda: MilvusTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
# ("pinecone-langchain", lambda: PineconeTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
# ("pinecone-llamaindex", lambda: PineconeTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
# ("pinecone-haystack", lambda: PineconeTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
("faiss-langchain", lambda: FaissTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("faiss-llamaindex", lambda: FaissTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
("faiss-haystack", lambda: FaissTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
("turbovec-langchain", lambda: TurbovecTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("turbovec-llamaindex", lambda: TurbovecTechnique.for_llamaindex(dataset, k), lambda r: LlamaIndexRAG(dataset, r)),
("turbovec-haystack", lambda: TurbovecTechnique.for_haystack(dataset, k), lambda r: HaystackRAG(dataset, r)),
]
test_retrievers(dataset, entries, "benchmark_results.csv", k=k)Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Dataset: 500003 documents & 20000 queries & 10 k
Building simlar-langchain...
Building simlar-llamaindex...
Building simlar-haystack...
Building chroma-langchain...
Building chroma-llamaindex...
Building chroma-haystack...
Building qdrant-langchain...
Building qdrant-llamaindex...
Building qdrant-haystack...
Building milvus-langchain...
skipped: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "SERVER: Received message larger than max (943897710 vs. 268435456)"
debug_error_string = "RESOURCE_EXHAUSTED:SERVER: Received message larger than max (943897710 vs. 268435456)"
>
Building milvus-llamaindex...
skipped: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "SERVER: Received message larger than max (943897710 vs. 268435456)"
debug_error_string = "RESOURCE_EXHAUSTED:SERVER: Received message larger than max (943897710 vs. 268435456)"
>
Building milvus-haystack...
skipped: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "SERVER: Received message larger than max (943897710 vs. 268435456)"
debug_error_string = "RESOURCE_EXHAUSTED:SERVER: Received message larger than max (943897710 vs. 268435456)"
>
Building faiss-langchain...
Building faiss-llamaindex...
Building faiss-haystack...
Building turbovec-langchain...
Building turbovec-llamaindex...
Building turbovec-haystack...
simlar-langchain: 20000/20000 queries (1 errors)
simlar-llamaindex: 20000/20000 queries (1 errors)
simlar-haystack: 20000/20000 queries (1 errors)
chroma-langchain: 20000/20000 queries
chroma-llamaindex: 20000/20000 queries
chroma-haystack: 20000/20000 queries
qdrant-langchain: 20000/20000 queries
qdrant-llamaindex: 20000/20000 queries
qdrant-haystack: 20000/20000 queries
faiss-langchain: 20000/20000 queries
faiss-llamaindex: 20000/20000 queries
faiss-haystack: 20000/20000 queries
turbovec-langchain: 20000/20000 queries
turbovec-llamaindex: 20000/20000 queries
turbovec-haystack: 20000/20000 queries
### LangChain
| Retriever | Hit@1 | Hit@10 | MRR | Build(ms) | Query(ms) |
|---|---|---|---|---|---|
| simlar | 29.7% | 85.6% | 0.472 | 146603 | 8.7 |
| chroma | 33.3% | 69.2% | 0.400 | 233203 | 2.8 |
| qdrant | 33.8% | 86.8% | 0.509 | 240039 | 403.9 |
| faiss | 33.9% | 86.8% | 0.509 | 136532 | 27.5 |
| turbovec | 33.9% | 86.8% | 0.509 | 131204 | 11.4 |
### LlamaIndex
| Retriever | Hit@1 | Hit@10 | MRR | Build(ms) | Query(ms) |
|---|---|---|---|---|---|
| simlar | 29.7% | 85.6% | 0.472 | 283577 | 8.8 |
| chroma | 33.3% | 69.2% | 0.400 | 241599 | 3.0 |
| qdrant | 33.8% | 86.8% | 0.509 | 236076 | 402.7 |
| faiss | 33.9% | 86.8% | 0.509 | 122314 | 16.9 |
| turbovec | 33.9% | 86.8% | 0.509 | 127805 | 11.3 |
### Haystack
| Retriever | Hit@1 | Hit@10 | MRR | Build(ms) | Query(ms) |
|---|---|---|---|---|---|
| simlar | 33.7% | 87.0% | 0.508 | 148513 | 9.7 |
| chroma | 33.3% | 69.2% | 0.400 | 244544 | 2.8 |
| qdrant | 33.8% | 86.8% | 0.509 | 237573 | 402.2 |
| faiss | 33.9% | 86.8% | 0.509 | 123991 | 17.6 |
| turbovec | 33.9% | 86.8% | 0.509 | 120694 | 11.4 |
Saved to benchmark_results.csv
Recall (Hit@10) vs. query latency, Haystack integration — the other two integrations tell the same story (see the tables above), so we're only charting one. The x-axis is broken on purpose: chroma and qdrant are true outliers pinned near the edges, while simlar, turbovec, and faiss — all within 8ms of each other — get a zoomed-in lane of their own so their gap is actually readable. simlar sits highest and furthest left in that lane: more recall and less latency than turbovec and faiss, both of which it strictly beats on both axes at once.
Reading the numbers
- simlar has the best recall-to-speed ratio of the five. Across all three integrations it queries in 8.7–9.7ms while landing within 1.2 points of the best recall in the table (85.6–87.0% Hit@10 vs. 86.8% for qdrant/faiss/turbovec) — and in its native Haystack integration, built on
SimlarHybridRetriever, it leads outright at 87.0% Hit@10, 0.508 MRR. Nothing else combines both dimensions as well: chroma queries faster but is 16–18 points behind on recall, qdrant and faiss match simlar's recall but take 2–46x longer per query, and only turbovec's 11.3–11.4ms comes close to simlar's speed. - Milvus couldn't complete the run. All three Milvus builds failed outright with a gRPC
RESOURCE_EXHAUSTEDerror: the single insert of 500K vectors (900MB+) exceeded Milvus Lite's default 256MB message limit. Every other backend ingested the same corpus without any tuning; Milvus would need batched inserts or a raised message-size ceiling just to participate at this scale. - Qdrant's query latency is the clear outlier. It answers in 402–404ms per integration — 15–140x slower than every other backend (chroma 2.8–3.0ms, simlar 8.7–9.7ms, turbovec 11.3–11.4ms, faiss 16.9–27.5ms), despite landing on the exact same recall as faiss and turbovec.
- Chroma is the sharpest speed-for-recall trade-off in the table. It's the fastest to query (2.8–3.0ms) but has by far the worst recall (69.2% Hit@10 vs. 85–87% for everything else) — over 500K passages, that gap means missing the right passage in roughly 3 of every 10 queries.
- qdrant, faiss, and turbovec tie. All three do plain nearest-neighbor search over the same embeddings (
all-MiniLM-L6-v2, cosine similarity) with no keyword component, so the top-k ranking for a given query is fixed by the embedding space itself, not by which engine computes the search — that's why all three land on exactly 86.8% Hit@10 and 0.509 MRR. What separates them is speed and footprint, not accuracy: turbovec builds fastest in two of the three integrations and queries in 11.3–11.4ms off 4-bit quantized vectors — slower than simlar or chroma, but faster than faiss's 16.9–27.5ms, for a fraction of the memory. - Pinecone had to be disabled for this run. Its calls are commented out in the benchmark entries (the class and its description are still in the notebook for reference) — upserting 500K vectors to a cloud index for every framework integration was too slow and ran into quota exhaustion.
Autotune
Helix with top_k, vector_k, and text_k set per query depth, using the best text_index (relevance or lookup), fusion weights, rrf_k, and BM25 k1/b found per retriever. In this example we are using only LangChain.
def helix(dataset: Dataset, k: int) -> LangchainBaseRetriever:
from simlar import HelixIndex, ReciprocalRankFusion, RelevanceIndex, SimlarEngine
from typing import ClassVar
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
corpus = dataset.corpus
vectors = model.encode(corpus, normalize_embeddings=True)
index = HelixIndex(
text_index=RelevanceIndex(k1=1.5, b=0.75),
vector_index=SimlarEngine(),
fusion=ReciprocalRankFusion(k=60, weights=[.1, .5]),
top_k=k, vector_k=max(50, 2 * k), text_k=200000,
)
index.add(ids=[str(i) for i in range(len(corpus))], texts=corpus, vectors=vectors)
class _R(LangchainBaseRetriever):
supports_batch: ClassVar[bool] = True
def _get_relevant_documents(self, query, *, run_manager=None):
qv = model.encode([query], normalize_embeddings=True)
hits = index.search(query_text=query, query_vector=qv, k=k, parallel=False)
return [Document(page_content=corpus[int(r.id)]) for r in hits]
def batch(self, queries, **kwargs):
qvs = model.encode(queries, normalize_embeddings=True)
all_hits = index.search(query_text=queries, query_vector=qvs, k=k, parallel=True, batch_size=10_000)
return [[Document(page_content=corpus[int(r.id)]) for r in hits] for hits in all_hits]
return _R()dataset = MsMarcoDataset.from_hf(n_passages=500_000, n_queries=20_000)
k = 100
entries = [
("simlar_default-langchain", lambda: SimlarTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("simlar_helix-langchain", lambda: helix(dataset, k), lambda r: LangchainRAG(dataset, r)),
("chroma-langchain", lambda: ChromaTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
# ("qdrant-langchain", lambda: QdrantTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("milvus-langchain", lambda: MilvusTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
# ("pinecone-langchain", lambda: PineconeTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("faiss-langchain", lambda: FaissTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
("turbovec-langchain", lambda: TurbovecTechnique.for_langchain(dataset, k), lambda r: LangchainRAG(dataset, r)),
]
test_retrievers(dataset, entries, "autotune_results.csv", k=k)Dataset: 500003 documents & 20000 queries & 100 k
Building simlar_default-langchain...
Building simlar_helix-langchain...
Building chroma-langchain...
Building milvus-langchain...
skipped: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "SERVER: Received message larger than max (943897710 vs. 268435456)"
debug_error_string = "RESOURCE_EXHAUSTED:SERVER: Received message larger than max (943897710 vs. 268435456)"
>
Building faiss-langchain...
Building turbovec-langchain...
simlar_default-langchain: 20000/20000 queries
simlar_helix-langchain: 20000/20000 queries
chroma-langchain: 20000/20000 queries
faiss-langchain: 20000/20000 queries
turbovec-langchain: 20000/20000 queries
### LangChain
| Retriever | Hit@1 | Hit@100 | MRR | Build(ms) | Query(ms) |
|---|---|---|---|---|---|
| simlar_default | 29.1% | 95.3% | 0.471 | 142099 | 7.0 |
| simlar_helix | 32.6% | 98.2% | 0.508 | 136919 | 2.7 |
| chroma | 33.4% | 95.4% | 0.442 | 241395 | 3.7 |
| faiss | 33.9% | 98.2% | 0.515 | 137405 | 28.5 |
| turbovec | 33.9% | 98.2% | 0.515 | 132794 | 11.5 |
Saved to autotune_results.csv
Recall (Hit@100) vs. query latency, autotuned Helix vs. the field — k raised to 100 and only the LangChain integration run this time. simlar_helix sits furthest left of the pack at 2.7ms, faster than even chroma's 3.7ms, while tying faiss and turbovec at the best recall in the table (98.2%). simlar_default, the same engine without the fusion, and chroma land at nearly the same recall (95.3% vs. 95.4%) — the fusion, not a faster vector engine, is what separates simlar_helix from the pack.
Takeaways
- Autotune gets there on its own. simlar_helix's fusion weights,
rrf_k, BM25k1/b, and per-depthtop_k/vector_k/text_kare all found automatically, not hand-picked — a simple, one-pass tuning step that's enough to turn simlar's own vector engine into the fastest, highest-recall retriever in the notebook. - simlar_helix leads on both axes that matter. 98.2% Hit@100 — tying faiss and turbovec for the best recall in the table — delivered in 2.7ms, the fastest query time of any retriever tested here: quicker than chroma's 3.7ms, and 4–10x quicker than turbovec and faiss.
- The fusion is what gets it there. simlar_default on its own reaches 95.3% Hit@100 in 7.0ms; autotuning the same engine into simlar_helix's BM25+RRF fusion lifts recall to 98.2% while cutting query time nearly 3x — the tuned fusion, built on simlar, is what puts it ahead of the field.
- chroma's speed doesn't buy it ranking quality. It queries fast (3.7ms) but its recall lands in the same 95%-ish band as the un-tuned simlar_default (95.4% vs. 95.3%), well short of simlar_helix's 98.2% — and its MRR (0.442) is the lowest in the entire table, even behind simlar_default's own 0.471. simlar_helix beats it outright on recall, ranking quality, and speed.
- qdrant and pinecone sat this one out on purpose. Both already ran in the k=10 benchmark above, where qdrant's 400ms+ query latency and pinecone's cloud round-trips were on full display — re-running either at k=100 over the same 500K-passage corpus would only add a long wait for results we'd already seen.
- Milvus still couldn't complete the run. Same gRPC
RESOURCE_EXHAUSTEDingest error as the k=10 benchmark — it would need batched inserts or a raised message-size ceiling to participate at this corpus size.
Choosing a retriever for your own RAG pipeline?
If retrieval accuracy or framework integration is the open question in your stack, we'd like to hear about your workload.
Talk to the TekDatum team →