sim_LAR Similarity Engine by TekDatum: Initial Performance & Rationale

Original paper: sim_LAR Similarity Engine by TekDatum — Initial Performance & Rationale, March 2025. Reproduced here in full.

At TekDatum, we are working on a new generation of similarity engines that can be optimized with existing relational indexes. This paper introduces a new index that delivers exciting results. We call this index sim_LAR (Large Augmented Retriever).

01Our Strategy

Traditionally, similarity search has provided the k-nearest neighbor query, the range query, and the k-nearest neighbor range query as the main API entry points. With the advent of Retrieval Augmented Generation, we thought deeply about the types of queries that could be provided to LLMs, and we concluded that there is an opportunity to create a similarity/vector search engine optimized for LLMs from its foundation. In the future, humans and machines will query similarity search engines in creative and complex ways, and it is our vision to create indexes that can support such use cases. We are also considering the guardrails that such similarity engines will require to prevent bad actors from obtaining or modifying information; our technology supports this.

Hybrid Search

Hybrid search blends two search methods: one for regular text and one for numerical data (called embeddings). It searches through a collection that has both types of information. Embeddings alone may miss important results, and this is why the community has concluded that keyword-based search using techniques such as BM25 can provide a helpful boost when combined with embeddings.

High-Dimensional Embeddings

Based on our experience building RAG solutions, high-dimensional embeddings (1024 dimensions or more) tend to perform better than smaller embeddings. The top 20 embeddings on the MTEB leaderboard have an average dimension of 3,900. High dimensionality can better capture information and improve the performance of NLP tasks.¹

Figure: Top 10 embedding approaches on the MTEB leaderboard.

1. docs.mistral.ai/capabilities/embeddings

02Search Quality

During some of our RAG deployments at TekDatum, we noticed that over different index creation cycles, chunks could disappear from query results. Our quality checks often failed when we had to append data into an index. This has a very simple explanation: similarity search as a field has given itself the permission to return approximate results in exchange for performance improvements. This, in the past, seemed reasonable — similarity search is such a hard problem that we were willing to provide approximate results that are good enough.

In the advent of RAG, approximate results are not enough. We need to be able to return perfect results so that LLMs can respond to queries in a consistent way. Therefore, our work focuses on perfect results or, at least, on measuring how big k needs to be for all relevant chunks to show up in a result set. Since machines are reading the k results, we are willing to allow a larger k than what traditional search engines have been able to use.

Why is this important? High-dimensional datasets are hard to search and index. It is even harder to obtain perfect results in high-dimensional spaces. On top of that, there are benefits of hybrid search, and this operation must be taken into account. All this is important because it sets our vision and direction. It helps us determine the kind of similarity search that LLMs and humans will need.

03sim_LAR Similarity Engine by TekDatum

Our sim_LAR index is designed from the ground up to handle high-dimensional objects and hybrid queries. We created a hybrid, high-dimensional index that is able to index and search faster than competing implementations while using less memory.

Fast creation

Creating and updating a similarity index quickly is important to maintain data that is as fresh as possible. Our engine is between 32 and 99 times faster than competing approaches.

Low memory

Similarity engines require many different structures internally, and therefore it is critical to reduce their size as much as possible. Our engine uses 2× to 6× less memory.

Fast search performance

Search time performance is the reason similarity search engines exist. Our engine is 1.26× to 8000× faster than commercial/open-source approaches.²

2. We did not include a comparison against weaviate because weaviate takes 82.5× more time to create the index and the results are 11.10× worse (they are not exact results when K=100).

Quality of results

Regarding quality — which we define as the maximum position of all the relevant chunks — our engine is 1.7× to 22× better than competing approaches.

04Benchmarks

In this section, we compare the sim_LAR engine against Pinecone, Weaviate, Qdrant, and Milvus. We chose these similarity search implementations because they are business-oriented and support hybrid search. Chroma and Zilliz were included in the underlying test runs and appear in the result charts below.

Datasets

PubMedQA. Sourced from PubMedQA: A Dataset for Biomedical Research Question Answering, this dataset contains expert-labeled biomedical question-answer pairs. While PubMedQA was originally smaller, our final biomedical dataset combines it with additional sources (gooaq, emrqa-msquad, MASHQA-JSON, medquad, medwiki) to total 8.8 million entries. This study, however, was focused on the questions and answers specifically from the PubMedQA dataset.

HotpotQA. A multi-hop question-answering dataset requiring reasoning across diverse documents. This benchmark tests systems' ability to retrieve and connect information from multiple sources.

Test definition

To test performance, we used a dataset consisting of 8,841,823 entries. We also established a ground truth of 10,000 queries from PubMedQA, which allowed us to validate the results effectively. For each query, the ground truth includes the three or four chunks that are able to answer the query effectively. We measure the ranking of the chunks, increasing k for each index until the chunk is returned, aiming for exact results and recording the position at which each index returns the corresponding value.

Max Position indicates the highest rank (position) of a relevant result in the list returned across all queries — lower values reflect better performance.

Our metrics focus on the typical use case of RAG similarity search: (1) the LLM needs to obtain all relevant results — we search for perfect accuracy; and (2) the LLM can read through multiple pages of results and find the most relevant one, so traditional evaluation metrics such as Recall@K may not be as helpful.

We used a machine equipped with 128 GB of system memory, 1 TB of SSD storage, and an AMD Ryzen Threadripper 2920X 12-core processor — equivalent to a g4dn.8xlarge or g5.8xlarge AWS instance, which costs between $1,400 and $1,500 per month.

05Results — PubMedQA Benchmark

Max Position performance, lower is better. Pinecone and Zilliz did not achieve a 100% retrieval rate for the chunks, while all other methods successfully returned the complete set of results. sim_LAR outperformed every other technique tested.

Index creation time. sim_LAR shows dramatic improvements when compared to other techniques.

Average search time per query, lower is better.

Average memory usage during searches. Pinecone does not report memory usage, but it cost us $900 to construct the index in this study for the 8.8M dataset — enough to reserve an EC2 instance with 1.5 TB of RAM for that amount per month. The Zilliz cost for the same dataset was $15.

06Results — HotpotQA Benchmark

Chroma results were omitted from this benchmark due to an incomplete execution.

Max Position performance, lower is better. Among all techniques tested, only sim_LAR V2 achieved a 100% retrieval rate for the chunks.

Index creation time. sim_LAR demonstrates significant improvements in efficiency compared to other techniques.

Average search time per query, lower is better.

Average memory usage during searches.

07Conclusions

The sim_LAR hybrid, high-dimensional similarity engine provides exceptional performance. We can search and index faster than competing approaches, using a fraction of the memory, with better quality results. sim_LAR shows non-trivial improvements on all these fronts. Our engine can considerably improve the performance, quality, and cost of RAG-based solutions.

After building sim_LAR V1, we learned a lot about what a similarity engine of the future should do. Based on these learnings, we continue to develop a next-generation similarity engine.