Index for ANN Search

Overview of Deep Lake's Index implementation for ANN search.

How Deep Lake Implements an Index for ANN Search

Deep Lake implements the Hierarchical Navigable Small World (HSNW) index for Approximate Nearest Neighbor (ANN) search. The index is based on the OSS Hsnwlib package with added optimizations. The implementation enables users to run queries on >35M embeddings in less than 1 second.

Unique aspects of Deep Lake's HSNW implementation

• Rapid index creation with multi-threading optimized for Deep Lake

• Efficient memory management that reduces RAM usage

Memory Management in Deep Lake

RAM Cost >> On-disk Cost >> Object Storage Cost

Minimizing RAM usage and maximizing object store significantly reduces costs of running a Vector Database. Deep Lake has a unique implementation of memory allocation that minimizes RAM requirement without any performance penalty: