Understanding the data layout in Deep Lake
Deep Lake uses a columnar storage architecture, and the columns in Deep Lake are referred to as
tensors. Data in the tensors can be added or modified, and the data in different tensors are independent of each other.
When data is appended to Deep Lake, certain important information is broken up and duplicated in a separate tensor, so that the information can be accessed and queried without loading all of the data. Examples include the shape of a sample (i.e. width, height, and number of channels for an image), or the metadata from file headers that were passed to
Deep Lake datasets and their tensors are indexed, and data at a given index that spans multiple tensors are referred to as
samples. Data at the same index are assumed to be related. For example, data in a
bboxtensor at index 100 is assumed to be related to data in the tensor
imageat index 100.
Most data in Deep Lake format is stored in
chunks, which are a blobs of data of a pre-defined size. The purpose of chunking is to accelerate the streaming of data across networks by increasing the amount of data that is transferred per network request.
Each tensors has its own chunks, and the default chunk size is 8MB. A single chunk consists of data from multiple indices when the individual data points (image, label, annotation, etc.) are smaller than the chunk size. Conversely, when individual data points are larger than the chunk size, the data is split among multiple chunks (tiling).
Exceptions to chunking logic are video data. Videos that are larger than the specified chunk size are not broken into smaller pieces, because Deep Lake uses efficient libraries to stream and access subsets of videos, thus making it unnecessary to split them apart.
Multiple tensor can be combined into
groups. Groups do not fundamentally change the way data is stored, but they are useful for helping Activeloop Platform understand how different tensors are related.
Deep Lake allows for ragged tensors (tensors of different length), so it is important to understand the terminology around dataset length:
- length (
len(ds)) - The length of the shortest tensor, as determined by its last index.
- minimum length (
ds.min_len) - Same as length
- minimum length (
ds.max_len) - The length of the longest tensor, as determined by its last index.
By default, Deep Lake throws an error if a tensor is accessed at an index at which data (empty or non-empty) has not been added. In the example below,
ds.bbox.numpy()would throw an error.
To pad the unspecified data and create a virtual view where the missing samples are treated as empty data, use
ds.max_view(). In the example below, the length of this virtual view would be 6.