Compute Engine

C++ part of the Deep Lake codebase optimized for faster data fetching and computations

Overview of Deep Lake's Optimized Compute Engine built in C++

In order to maximize performance of compute-heavy Deep Lake's features, such as distributed dataloading or large queries, it is necessary to move beyond their pure-Python OSS implementations and use the C++ codebase in the Deep Lake Compute Engine.

The interface for the Compute Engine is still in Python, and it is installed using pip install "deeplake[enterprise]", which delivers the compiled C++ code from binaries.

In order to use the optimized Compute Engine, datasets can be stored in the user's cloud, but they must be connected to Deep Lake using Managed Credentials.

Features in the Optimized Compute Engine

pageQuerying DatasetspagePerformant Dataloader