Inside the sim_LAR RAG Retrieval Benchmark Notebook

The following is the full notebook behind our RAG retrieval benchmark — every markdown cell and every line of code, in the order it ran, along with the actual output.

All four notebooks are public, so you can run them yourself. This post walks through 10_rags_benchmark.ipynb, which benchmarks sim_LAR through all three framework integrations at once — but each integration also has its own dedicated notebook:

10_rags_benchmark.ipynb — the combined benchmark covered in this post.
07_langchain_benchmark.ipynb — LangChain integration.
08_llamaindex_benchmark.ipynb — LlamaIndex integration.
09_haystack_benchmark.ipynb — Haystack integration.

10 — RAG Retriever Benchmark

This notebook evaluates simlar against other vector stores in a retrieval-augmented generation (RAG) setting. We measure how well each retriever finds the correct document for a given question — the core task in RAG before any LLM is involved.

What we cover

Loading a question-answering dataset and building a ground-truth corpus
Evaluating retrieval quality with Hit@1, Hit@3, and MRR
Comparing simlar against Chroma, Qdrant, and Milvus
Measuring index build time and query latency

%pip install -q \
    sentence-transformers \
    langchain-core langchain-huggingface langchain-chroma langchain-community langchain-deepseek \
    qdrant-client \
    pymilvus \
    pinecone \
    llama-index-core llama-index-embeddings-huggingface llama-index-llms-openai-like \
    haystack-ai

Environment variables

API keys for Pinecone and DeepSeek. Each prompt only appears if the key is not already set in the environment.

import os
from getpass import getpass


if not os.environ.get("PINECONE_API_KEY"):
    os.environ["PINECONE_API_KEY"] = getpass("Pinecone API key: ")
if not os.environ.get("DEEPSEEK_API_KEY"):
    os.environ["DEEPSEEK_API_KEY"] = getpass("DeepSeek API key: ")

Dataset

We use Kaggle — a reading comprehension dataset where each question has a corresponding context passage that contains the answer. This structure maps naturally to RAG: the context passages become the corpus and the questions become the queries. For each query we know exactly which document should be retrieved, giving us a ground truth to evaluate against. Place the CSV file in the same directory as this notebook before running.

import csv
from abc import ABC, abstractmethod


class Dataset(ABC):
    corpus: list[str]
    queries: list[str]

    @abstractmethod
    def evaluate(self, query: str, docs: list[str]) -> bool: ...

    @abstractmethod
    def rank(self, query: str, docs: list[str]) -> int | None: ...


class SquadDataset(Dataset):
    def __init__(self, corpus, queries, query_to_context):
        self.corpus = corpus
        self.queries = queries
        self._query_to_context = query_to_context

    @classmethod
    def from_csv(cls, path, n_questions=None, n_contexts=None):
        seen, corpus, queries, query_to_context = set(), [], [], {}
        with open(path) as f:
            for row in csv.DictReader(f):
                if n_questions is not None and len(queries) == n_questions:
                    break
                ctx = row["context"]
                if ctx not in seen:
                    if n_contexts is not None and len(corpus) >= n_contexts:
                        continue
                    seen.add(ctx)
                    corpus.append(ctx)
                queries.append(row["question"])
                query_to_context[row["question"]] = ctx
        return cls(corpus=corpus, queries=queries, query_to_context=query_to_context)

    def evaluate(self, query, docs):
        return self._query_to_context.get(query) in docs

    def rank(self, query: str, docs: list[str]) -> int | None:
        correct = self._query_to_context.get(query)
        for i, doc in enumerate(docs):
            if doc == correct:
                return i
        return None

Backends

Each backend wraps a vector store and exposes a uniform interface: given a query string, return the top-k most relevant documents from the corpus. We test simlar across its three integration layers — LangChain, LlamaIndex, and Haystack — and compare against Chroma, Qdrant, and Milvus.

All backend use the same embedding model (all-MiniLM-L6-v2) so that differences in retrieval quality reflect the underlying index, not the embeddings.

from typing import Protocol

class HaystackBaseRetriever(Protocol):
    def run(self, query: str) -> dict: ...

simlar

simlar exposes a hybrid index that combines keyword (BM25) and semantic (vector) search. It integrates natively with LangChain, LlamaIndex, and Haystack — the three classes below each wrap the same simlar index through a different integration layer.

from langchain_core.documents import Document
from langchain_core.retrievers import BaseRetriever as LangchainBaseRetriever
from llama_index.core.retrievers import BaseRetriever as LlamaBaseRetriever
from llama_index.core.schema import NodeWithScore, TextNode, QueryBundle


class SimlarBackend:
    @classmethod
    def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
        from langchain_huggingface import HuggingFaceEmbeddings
        from simlar.integrations.langchain.simlar_vector_store import SimlarVectorStore
        from simlar.integrations.langchain.langchain_retriever import SimlarRetriever
        SimlarRetriever.model_rebuild()
        store = SimlarVectorStore.from_texts(
            texts=dataset.corpus,
            embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
        )
        return SimlarRetriever(vector_store=store, k=k)

    @classmethod
    def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
        import numpy as np
        from llama_index.embeddings.huggingface import HuggingFaceEmbedding
        from simlar.integrations.llama_index.simlar_retriever import SimlarRetriever
        embed_model = HuggingFaceEmbedding(model_name="all-MiniLM-L6-v2")
        vectors = np.array(embed_model.get_text_embedding_batch(dataset.corpus), dtype=np.float32)
        return SimlarRetriever.from_texts(
            texts=dataset.corpus,
            ids=[str(i) for i in range(len(dataset.corpus))],
            vectors=vectors,
            embed_model=embed_model,
            k=k,
        )

    @classmethod
    def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
        from haystack import Document as HaystackDocument, component
        from haystack.components.embedders import (
            SentenceTransformersDocumentEmbedder,
            SentenceTransformersTextEmbedder,
        )
        from simlar.integrations.haystack.simlar_document_store import SimlarDocumentStore
        from simlar.integrations.haystack.simlar_retriever import SimlarHybridRetriever

        store = SimlarDocumentStore(top_k=k)
        doc_embedder = SentenceTransformersDocumentEmbedder(model="sentence-transformers/all-MiniLM-L6-v2", progress_bar=False)
        doc_embedder.warm_up()
        store.write_documents(
            doc_embedder.run([HaystackDocument(content=t) for t in dataset.corpus])["documents"]
        )
        _retriever = SimlarHybridRetriever(document_store=store, top_k=k)
        _text_embedder = SentenceTransformersTextEmbedder(model="sentence-transformers/all-MiniLM-L6-v2", progress_bar=False)
        _text_embedder.warm_up()

        @component
        class _R:
            @component.output_types(documents=list[HaystackDocument])
            def run(self, query: str) -> dict:
                embedding = _text_embedder.run(query)["embedding"]
                return _retriever.run(query=query, query_embedding=embedding)

        return _R()

Chroma

Chroma is an open-source embedding database designed for AI applications. It stores vectors in memory or on disk and supports similarity search out of the box. In this benchmark we use it in in-memory mode so there is no persistence overhead between runs.

class ChromaBackend:
    @classmethod
    def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
        from langchain_chroma import Chroma
        from langchain_huggingface import HuggingFaceEmbeddings
        store = Chroma.from_texts(
            texts=dataset.corpus,
            embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
        )
        return store.as_retriever(search_kwargs={"k": k})

    @classmethod
    def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
        from langchain_chroma import Chroma
        from langchain_huggingface import HuggingFaceEmbeddings
        _store = Chroma.from_texts(
            texts=dataset.corpus,
            embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
        )

        class _R(LlamaBaseRetriever):
            def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
                return [NodeWithScore(node=TextNode(text=d.page_content))
                        for d in _store.similarity_search(query_bundle.query_str, k=k)]

        return _R()

    @classmethod
    def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
        from haystack import Document as HaystackDocument, component
        from langchain_chroma import Chroma
        from langchain_huggingface import HuggingFaceEmbeddings
        _store = Chroma.from_texts(
            texts=dataset.corpus,
            embedding=HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2"),
        )

        @component
        class _R:
            @component.output_types(documents=list[HaystackDocument])
            def run(self, query: str) -> dict:
                return {"documents": [HaystackDocument(content=d.page_content)
                                      for d in _store.similarity_search(query, k=k)]}

        return _R()

Qdrant

Qdrant is a vector search engine built in Rust, optimized for high-performance similarity search. It supports filtering, payloads, and multiple distance metrics. Here we run it in in-memory mode (":memory:") to keep setup simple and avoid disk I/O in the benchmark.

class QdrantBackend:
    @classmethod
    def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
        from sentence_transformers import SentenceTransformer
        from qdrant_client import QdrantClient
        from qdrant_client.models import Distance, VectorParams, PointStruct
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        _client = QdrantClient(":memory:")
        _client.create_collection(
            collection_name="corpus",
            vectors_config=VectorParams(size=vectors.shape[1], distance=Distance.COSINE),
        )
        _client.upsert(
            collection_name="corpus",
            points=[PointStruct(id=i, vector=vectors[i].tolist(), payload={"text": text})
                    for i, text in enumerate(_corpus)],
        )

        class _R(LangchainBaseRetriever):
            def _get_relevant_documents(self, query, *, run_manager=None):
                qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
                return [Document(page_content=r.payload["text"])
                        for r in _client.query_points(collection_name="corpus", query=qv, limit=k).points]

        return _R()

    @classmethod
    def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
        from sentence_transformers import SentenceTransformer
        from qdrant_client import QdrantClient
        from qdrant_client.models import Distance, VectorParams, PointStruct
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        _client = QdrantClient(":memory:")
        _client.create_collection(
            collection_name="corpus",
            vectors_config=VectorParams(size=vectors.shape[1], distance=Distance.COSINE),
        )
        _client.upsert(
            collection_name="corpus",
            points=[PointStruct(id=i, vector=vectors[i].tolist(), payload={"text": text})
                    for i, text in enumerate(_corpus)],
        )

        class _R(LlamaBaseRetriever):
            def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
                qv = _model.encode([query_bundle.query_str], normalize_embeddings=True)[0].tolist()
                results = _client.query_points(collection_name="corpus", query=qv, limit=k).points
                return [NodeWithScore(node=TextNode(text=r.payload["text"]), score=r.score) for r in results]

        return _R()

    @classmethod
    def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
        from haystack import Document as HaystackDocument, component
        from sentence_transformers import SentenceTransformer
        from qdrant_client import QdrantClient
        from qdrant_client.models import Distance, VectorParams, PointStruct
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        _client = QdrantClient(":memory:")
        _client.create_collection(
            collection_name="corpus",
            vectors_config=VectorParams(size=vectors.shape[1], distance=Distance.COSINE),
        )
        _client.upsert(
            collection_name="corpus",
            points=[PointStruct(id=i, vector=vectors[i].tolist(), payload={"text": text})
                    for i, text in enumerate(_corpus)],
        )

        @component
        class _R:
            @component.output_types(documents=list[HaystackDocument])
            def run(self, query: str) -> dict:
                qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
                results = _client.query_points(collection_name="corpus", query=qv, limit=k).points
                return {"documents": [HaystackDocument(content=r.payload["text"]) for r in results]}

        return _R()

Milvus

Milvus is a distributed vector database built for large-scale similarity search. We use Milvus Lite — a lightweight version that runs locally as a file-based database, with no server required. Note that only one process can open the database file at a time.

class MilvusBackend:
    @classmethod
    def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
        from sentence_transformers import SentenceTransformer
        from pymilvus import MilvusClient
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        _client = MilvusClient("/tmp/milvus_rag.db")
        if _client.has_collection("corpus"):
            _client.drop_collection("corpus")
        _client.create_collection(collection_name="corpus", dimension=vectors.shape[1])
        _client.insert(
            collection_name="corpus",
            data=[{"id": i, "vector": vectors[i].tolist(), "text": text}
                  for i, text in enumerate(_corpus)],
        )

        class _R(LangchainBaseRetriever):
            def _get_relevant_documents(self, query, *, run_manager=None):
                qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
                results = _client.search(collection_name="corpus", data=[qv], limit=k, output_fields=["text"])
                return [Document(page_content=r["entity"]["text"]) for r in results[0]]

        return _R()

    @classmethod
    def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
        from sentence_transformers import SentenceTransformer
        from pymilvus import MilvusClient
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        _client = MilvusClient("/tmp/milvus_rag.db")
        if _client.has_collection("corpus"):
            _client.drop_collection("corpus")
        _client.create_collection(collection_name="corpus", dimension=vectors.shape[1])
        _client.insert(
            collection_name="corpus",
            data=[{"id": i, "vector": vectors[i].tolist(), "text": text}
                  for i, text in enumerate(_corpus)],
        )

        class _R(LlamaBaseRetriever):
            def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
                qv = _model.encode([query_bundle.query_str], normalize_embeddings=True)[0].tolist()
                results = _client.search(collection_name="corpus", data=[qv], limit=k, output_fields=["text"])
                return [NodeWithScore(node=TextNode(text=r["entity"]["text"])) for r in results[0]]

        return _R()

    @classmethod
    def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
        from haystack import Document as HaystackDocument, component
        from sentence_transformers import SentenceTransformer
        from pymilvus import MilvusClient
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        _client = MilvusClient("/tmp/milvus_rag.db")
        if _client.has_collection("corpus"):
            _client.drop_collection("corpus")
        _client.create_collection(collection_name="corpus", dimension=vectors.shape[1])
        _client.insert(
            collection_name="corpus",
            data=[{"id": i, "vector": vectors[i].tolist(), "text": text}
                  for i, text in enumerate(_corpus)],
        )

        @component
        class _R:
            @component.output_types(documents=list[HaystackDocument])
            def run(self, query: str) -> dict:
                qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
                results = _client.search(collection_name="corpus", data=[qv], limit=k, output_fields=["text"])
                return {"documents": [HaystackDocument(content=r["entity"]["text"]) for r in results[0]]}

        return _R()

Pinecone

Pinecone is a managed vector database — unlike the others, it runs as an external cloud service. This means build time includes network latency for uploading vectors, and query time includes a round-trip to Pinecone's servers. A PINECONE_API_KEY environment variable is required.

class PineconeBackend:
    @classmethod
    def for_langchain(cls, dataset: Dataset, k: int = 3) -> LangchainBaseRetriever:
        from sentence_transformers import SentenceTransformer
        from pinecone import Pinecone, ServerlessSpec
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        pc = Pinecone()
        if "rag-corpus" not in [i.name for i in pc.list_indexes()]:
            pc.create_index(
                name="rag-corpus",
                dimension=vectors.shape[1],
                metric="cosine",
                spec=ServerlessSpec(cloud="aws", region="us-east-1"),
            )
        _index = pc.Index("rag-corpus")
        batch_size = 100
        records = [{"id": str(i), "values": vectors[i].tolist(), "metadata": {"text": text}}
                   for i, text in enumerate(_corpus)]
        for i in range(0, len(records), batch_size):
            _index.upsert(vectors=records[i:i + batch_size])

        class _R(LangchainBaseRetriever):
            def _get_relevant_documents(self, query, *, run_manager=None):
                qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
                results = _index.query(vector=qv, top_k=k, include_metadata=True)
                return [Document(page_content=m["metadata"]["text"]) for m in results["matches"]]

        return _R()

    @classmethod
    def for_llamaindex(cls, dataset: Dataset, k: int = 3) -> LlamaBaseRetriever:
        from sentence_transformers import SentenceTransformer
        from pinecone import Pinecone, ServerlessSpec
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        pc = Pinecone()
        if "rag-corpus" not in [i.name for i in pc.list_indexes()]:
            pc.create_index(
                name="rag-corpus",
                dimension=vectors.shape[1],
                metric="cosine",
                spec=ServerlessSpec(cloud="aws", region="us-east-1"),
            )
        _index = pc.Index("rag-corpus")
        batch_size = 100
        records = [{"id": str(i), "values": vectors[i].tolist(), "metadata": {"text": text}}
                   for i, text in enumerate(_corpus)]
        for i in range(0, len(records), batch_size):
            _index.upsert(vectors=records[i:i + batch_size])

        class _R(LlamaBaseRetriever):
            def _retrieve(self, query_bundle: QueryBundle) -> list[NodeWithScore]:
                qv = _model.encode([query_bundle.query_str], normalize_embeddings=True)[0].tolist()
                results = _index.query(vector=qv, top_k=k, include_metadata=True)
                return [NodeWithScore(node=TextNode(text=m["metadata"]["text"])) for m in results["matches"]]

        return _R()
    
    @classmethod
    def for_haystack(cls, dataset: Dataset, k: int = 3) -> HaystackBaseRetriever:
        from haystack import Document as HaystackDocument, component
        from sentence_transformers import SentenceTransformer
        from pinecone import Pinecone, ServerlessSpec
        _model = SentenceTransformer("all-MiniLM-L6-v2")
        _corpus = dataset.corpus
        vectors = _model.encode(_corpus, normalize_embeddings=True)
        pc = Pinecone()
        if "rag-corpus" not in [i.name for i in pc.list_indexes()]:
            pc.create_index(
                name="rag-corpus",
                dimension=vectors.shape[1],
                metric="cosine",
                spec=ServerlessSpec(cloud="aws", region="us-east-1"),
            )
        _index = pc.Index("rag-corpus")
        batch_size = 100
        records = [{"id": str(i), "values": vectors[i].tolist(), "metadata": {"text": text}}
                   for i, text in enumerate(_corpus)]
        for i in range(0, len(records), batch_size):
            _index.upsert(vectors=records[i:i + batch_size])

        @component
        class _R:
            @component.output_types(documents=list[HaystackDocument])
            def run(self, query: str) -> dict:
                qv = _model.encode([query], normalize_embeddings=True)[0].tolist()
                results = _index.query(vector=qv, top_k=k, include_metadata=True)
                return {"documents": [HaystackDocument(content=m["metadata"]["text"]) for m in results["matches"]]}

        return _R()

RAG classes

Each RAG class wraps a retriever and exposes two methods:

retrieve(query) — returns the top-k documents as strings, no LLM involved. Used for the benchmark.
ask(query) — runs the full RAG pipeline with an LLM and returns an answer.

from dataclasses import dataclass


@dataclass
class AskResult:
    question: str
    answer: str | None
    docs: list[str]
    hit: bool


class RAG:
    def retrieve(self, query: str) -> list[str]: ...
    def ask(self, query: str) -> AskResult: ...

LangChain RAG

Wraps a LangChain retriever. retrieve() calls the retriever directly. ask() runs the full chain with an LLM.

from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough, RunnableParallel
from langchain_deepseek import ChatDeepSeek


class LangchainRAG(RAG):
    def __init__(self, dataset: Dataset, retriever: LangchainBaseRetriever):
        self._dataset = dataset
        self._retriever = retriever
        prompt = ChatPromptTemplate.from_template(
            "Answer using only the context below. Be concise.\n\n"
            "Context:\n{context}\n\nQuestion: {question}"
        )
        self._chain = (
            RunnableParallel(docs=retriever, question=RunnablePassthrough())
            | RunnablePassthrough.assign(context=lambda x: "\n\n".join(d.page_content for d in x["docs"]))
            | RunnableParallel(
                answer=prompt | ChatDeepSeek(model="deepseek-chat") | StrOutputParser(),
                docs=lambda x: x["docs"],
            )
        )

    def retrieve(self, query: str) -> list[str]:
        return [d.page_content for d in self._retriever.invoke(query)]

    def ask(self, query: str) -> AskResult:
        result = self._chain.invoke(query)
        docs = [d.page_content for d in result["docs"]]
        return AskResult(question=query, answer=result["answer"], docs=docs,
                         hit=self._dataset.evaluate(query, docs))

LlamaIndex RAG

Wraps a LlamaIndex retriever. retrieve() calls the retriever directly. ask() runs the full query engine with an LLM.

from llama_index.core.retrievers import BaseRetriever as LlamaBaseRetriever


class LlamaIndexRAG(RAG):
    def __init__(self, dataset: Dataset, retriever: LlamaBaseRetriever):
        self._dataset = dataset
        self._retriever = retriever
        from llama_index.core import Settings
        from llama_index.core.query_engine import RetrieverQueryEngine
        from llama_index.embeddings.huggingface import HuggingFaceEmbedding
        from llama_index.llms.openai_like import OpenAILike
        Settings.embed_model = HuggingFaceEmbedding(model_name="all-MiniLM-L6-v2")
        Settings.llm = OpenAILike(
            model="deepseek-chat",
            api_base="https://api.deepseek.com/v1",
            api_key=__import__("os").environ["DEEPSEEK_API_KEY"],
            context_window=64000,
            is_chat_model=True,
        )
        self._query_engine = RetrieverQueryEngine.from_args(retriever=retriever)

    def retrieve(self, query: str) -> list[str]:
        return [n.node.text for n in self._retriever.retrieve(query)]

    def ask(self, query: str) -> AskResult:
        response = self._query_engine.query(query)
        docs = [n.node.text for n in response.source_nodes]
        return AskResult(question=query, answer=str(response), docs=docs,
                         hit=self._dataset.evaluate(query, docs))

Haystack RAG

Wraps a Haystack retriever. retrieve() calls the retriever directly. ask() runs a Haystack pipeline with an LLM.

class HaystackRAG(RAG):
    def __init__(self, dataset: Dataset, retriever):
        self._dataset = dataset
        self._retriever = retriever
        from haystack import Pipeline
        from haystack.components.builders import PromptBuilder
        from haystack.components.generators.chat import OpenAIChatGenerator
        from haystack.utils import Secret
        self._pipeline = Pipeline()
        self._pipeline.add_component("prompt", PromptBuilder(
            template="Answer using only the context below. Be concise.\n\nContext:\n{% for d in documents %}{{ d.content }}\n{% endfor %}\nQuestion: {{ query }}",
            required_variables=["documents", "query"],
        ))
        self._pipeline.add_component("generator", OpenAIChatGenerator(
            model="deepseek-chat",
            api_base_url="https://api.deepseek.com",
            api_key=Secret.from_env_var("DEEPSEEK_API_KEY"),
        ))
        self._pipeline.connect("prompt.prompt", "generator.messages")

    def retrieve(self, query: str) -> list[str]:
        return [d.content for d in self._retriever.run(query)["documents"]]

    def ask(self, query: str) -> AskResult:
        docs = self._retriever.run(query)["documents"]
        result = self._pipeline.run({"prompt": {"documents": docs, "query": query}})
        answer = result["generator"]["replies"][0].text
        doc_texts = [d.content for d in docs]
        return AskResult(question=query, answer=answer, docs=doc_texts,
                         hit=self._dataset.evaluate(query, doc_texts))

Benchmark

test_retrievers runs the full benchmark in two phases:

1. Build — each retriever indexes the corpus and the elapsed time is recorded.

2. Query — every question in the dataset is passed to rag.retrieve() and the result is evaluated against the ground truth.

Metrics

Hit@1 — fraction of queries where the correct document is ranked first.
Hit@3 — fraction of queries where the correct document appears in the top 3.
MRR (Mean Reciprocal Rank) — average of 1/rank for the correct document. Penalizes results that are correct but ranked lower.

import time
from typing import Callable
import contextlib
import io

@contextlib.contextmanager
def _suppress():
    devnull = os.open(os.devnull, os.O_WRONLY)
    old_stdout = os.dup(1)
    old_stderr = os.dup(2)
    os.dup2(devnull, 1)
    os.dup2(devnull, 2)
    os.close(devnull)
    try:
        with contextlib.redirect_stdout(io.StringIO()):
            with contextlib.redirect_stderr(io.StringIO()):
                yield
    finally:
        os.dup2(old_stdout, 1)
        os.dup2(old_stderr, 2)
        os.close(old_stdout)
        os.close(old_stderr)

def test_retrievers(dataset: SquadDataset, out:str, k: int = 3) -> None:
    print(f"Dataset: {len(dataset.corpus)} documents & {len(dataset.queries)} queries")

    entries: list[tuple[str, Callable[[], object], Callable[[object], RAG]]] = [
        ("simlar-langchain",    lambda: SimlarBackend.for_langchain(dataset, k),          lambda r: LangchainRAG(dataset, r)),
        ("simlar-llama",        lambda: SimlarBackend.for_llamaindex(dataset, k),         lambda r: LlamaIndexRAG(dataset, r)),
        ("simlar-haystack",     lambda: SimlarBackend.for_haystack(dataset, k),           lambda r: HaystackRAG(dataset, r)),
        ("chroma-langchain",    lambda: ChromaBackend.for_langchain(dataset, k),          lambda r: LangchainRAG(dataset, r)),
        ("chroma-llama",        lambda: ChromaBackend.for_llamaindex(dataset, k),         lambda r: LlamaIndexRAG(dataset, r)),
        ("chroma-haystack",     lambda: ChromaBackend.for_haystack(dataset, k),           lambda r: HaystackRAG(dataset, r)),
        ("qdrant-langchain",    lambda: QdrantBackend.for_langchain(dataset, k),          lambda r: LangchainRAG(dataset, r)),
        ("qdrant-llama",        lambda: QdrantBackend.for_llamaindex(dataset, k),         lambda r: LlamaIndexRAG(dataset, r)),
        ("qdrant-haystack",     lambda: QdrantBackend.for_haystack(dataset, k),           lambda r: HaystackRAG(dataset, r)),
        ("milvus-langchain",    lambda: MilvusBackend.for_langchain(dataset, k),          lambda r: LangchainRAG(dataset, r)),
        ("milvus-llama",        lambda: MilvusBackend.for_llamaindex(dataset, k),         lambda r: LlamaIndexRAG(dataset, r)),
        ("milvus-haystack",     lambda: MilvusBackend.for_haystack(dataset, k),           lambda r: HaystackRAG(dataset, r)),
        ("pinecone-langchain",  lambda: PineconeBackend.for_langchain(dataset, k),        lambda r: LangchainRAG(dataset, r)),
        ("pinecone-llama",      lambda: PineconeBackend.for_llamaindex(dataset, k),       lambda r: LlamaIndexRAG(dataset, r)),
        ("pinecone-haystack",   lambda: PineconeBackend.for_haystack(dataset, k),         lambda r: HaystackRAG(dataset, r)),
    ]

    # Build phase
    rags: list[dict] = []
    for name, retriever_factory, rag_factory in entries:
        print(f"Building {name}...")
        with _suppress():
            t0 = time.perf_counter()
            retriever = retriever_factory()
            build_ms = (time.perf_counter() - t0) * 1000
            rag = rag_factory(retriever)
        rags.append({"name": name, "rag": rag, "build_ms": build_ms})

    # Query phase
    n = len(dataset.queries)
    for entry in rags:
        name, rag = entry["name"], entry["rag"]
        hit1 = hitk = mrr = 0.0
        errors = 0
        t0 = time.perf_counter()
        for i, q in enumerate(dataset.queries, 1):
            try:
                with _suppress():
                    docs = rag.retrieve(q)
                    r = dataset.rank(q, docs)
                if r is not None:
                    if r == 0: hit1 += 1
                    hitk += 1
                    mrr += 1 / (r + 1)
            except Exception as e:
                errors += 1
                with open("errors.txt", "a") as f:
                    f.write(f"[{name}] query={repr(q)} error={e}\n")
            if i % 100 == 0 or i == n:
                print(f"\r  {name}: {i}/{n} queries{f' ({errors} errors)' if errors else ''}", end="", flush=True)
        print()
        entry["query_ms"] = (time.perf_counter() - t0) * 1000 / n
        entry["hit1"] = hit1 / n
        entry["hitk"] = hitk / n
        entry["mrr"]  = mrr  / n

    # Results
    print(f"\n{'Retriever':<30} {'Hit@1':>6} {'Hit@3':>6} {'MRR':>6} {'Build(ms)':>10} {'Query(ms)':>10}")
    print("-" * 76)
    for e in rags:
        print(f"{e['name']:<30} {e['hit1']:>6.1%} {e['hitk']:>6.1%} {e['mrr']:>6.3f} {e['build_ms']:>10.0f} {e['query_ms']:>10.1f}")
    
    # Save to CSV
    with open(out, "w", newline="") as f:
        writer = csv.DictWriter(f, fieldnames=["name", "hit1", "hitk", "mrr", "build_ms", "query_ms"])
        writer.writeheader()
        for e in rags:
            writer.writerow({k: e[k] for k in writer.fieldnames})
    print(f"Saved to {out}")

squad_dataset = SquadDataset.from_csv("SQuAD-v1.2.csv", n_questions=1000)
test_retrievers(squad_dataset, "benchmark_results.csv", k=3)

Dataset: 119 documents & 1000 queries
Building simlar-langchain...
Building simlar-llama...
Building simlar-haystack...
Building chroma-langchain...
Building chroma-llama...
Building chroma-haystack...
Building qdrant-langchain...
Building qdrant-llama...
Building qdrant-haystack...
Building milvus-langchain...
Building milvus-llama...
Building milvus-haystack...
Building pinecone-langchain...
Building pinecone-llama...
Building pinecone-haystack...
  simlar-langchain: 1000/1000 queries
  simlar-llama: 1000/1000 queries
  simlar-haystack: 1000/1000 queries
  chroma-langchain: 1000/1000 queries
  chroma-llama: 1000/1000 queries
  chroma-haystack: 1000/1000 queries
  qdrant-langchain: 1000/1000 queries
  qdrant-llama: 1000/1000 queries
  qdrant-haystack: 1000/1000 queries
  milvus-langchain: 1000/1000 queries
  milvus-llama: 1000/1000 queries
  milvus-haystack: 1000/1000 queries
  pinecone-langchain: 1000/1000 queries
  pinecone-llama: 1000/1000 queries
  pinecone-haystack: 1000/1000 queries

Retriever                       Hit@1  Hit@3    MRR  Build(ms)  Query(ms)
----------------------------------------------------------------------------
simlar-langchain                70.1%  90.7%  0.794       1617        4.5
simlar-llama                    70.1%  90.7%  0.794       2083        6.0
simlar-haystack                 59.3%  79.7%  0.683         78        5.5
chroma-langchain                61.2%  61.2%  0.612       1619        2.5
chroma-llama                    61.2%  61.2%  0.612       1643        2.6
chroma-haystack                 61.2%  61.2%  0.612       1553        2.5
qdrant-langchain                61.3%  81.2%  0.702       1544        2.2
qdrant-llama                    61.3%  81.2%  0.702       1555        2.3
qdrant-haystack                 61.3%  81.2%  0.702       1447        2.3
milvus-langchain                61.3%  81.2%  0.702       2228       12.0
milvus-llama                    61.3%  81.2%  0.702       1980       13.0
milvus-haystack                 61.3%  81.2%  0.702       1977       12.5
pinecone-langchain              55.6%  75.6%  0.646       2700       81.8
pinecone-llama                  55.6%  75.6%  0.646       2834       79.8
pinecone-haystack               55.6%  75.6%  0.646       2583       82.3
Saved to benchmark_results.csv

Choosing a retriever for your own RAG pipeline?

If retrieval accuracy or framework integration is the open question in your stack, we'd like to hear about your workload.

Talk to the TekDatum team →