Querying Datasets
Activeloop Platform offer a highly-performant SQL-style query engine for filtering your data

How to query machine learning datasets using Activeloop's query engine

Querying datasets is a critical aspect of data science workflows that enables users to filter datasets and focus their work on the most relevant data at hand. Activeloop offers a highly-performant dataset query engine built in C++ and optimized for Deep Lake datasets.

Dataset Query Summary

Saving and utilizing dataset query results

The query results (Dataset Views) can be saved in the UI as shown above, or if the view is generated in Python, it can be saved using the Python API below. Full details are available here.
ds_view.save_view(message = 'Samples with monarchs')
In order to maintain data lineage, Dataset Views are immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the HEAD.
Dataset Views can be loaded in the python API and they can passed to ML frameworks just like regular datasets:
ds_view = ds.load_view(view_id, optimize = True, num_workers = 2)
for data in ds_view.pytorch():
# Training loop here
The optimize parameter in ds.load_view(..., optimize = True) materializes the Dataset View into a new sub-dataset that is optimized for streaming. If the original dataset uses linked tensors, the data will be copied to Deep Lake format.
Optimizing the Dataset View is critical for achieving rapid streaming.
If the saved Dataset View is no longer needed, it can be deleted using:
ds.delete_view(view_id)

Dataset Query Syntax

CONTAINS and ==

# Exact match, which generally requires that the sample
# has 1 value, i.e. no lists or multi-dimensional arrays
select * where tensor_name == 'text_value' # If value is numeric
select * where tensor_name == numeric_value # If values is text
select * where contains(tensor_name, 'text_value')
Any special characters in tensor or group names should be wrapped with double-quotes:
select * where contains("tensor-name", 'text_value')
select * where "tensor_name/group_name" == numeric_value

SHAPE

select * where shape(tensor_name)[dimension_index] > numeric_value
select * where shape(tensor_name)[1] > numeric_value # Second array dimension > value

LIMIT

select * where contains(tensor_name, 'text_value') limit num_samples

AND and OR

select * where contains(tensor_name, 'text_value') and contains(tensor_name_2, numeric_value)
select * where contains(tensor_name, 'text_value') or tensor_name_2 == numeric_value
select * where (contains(tensor_name, 'text_value') and shape(tensor_name_2)[dimension_index]>numeric_value) or contains(tensor_name, 'text_value_2')

NOT

UNION and INTERSECT

(select * where contains(tensor_name, 'value')) intersect (select * where contains(tensor_name, 'value_2'))
(select * where contains(tensor_name, 'value') limit 100) union (select * where shape(tensor_name)[0] > numeric_value limit 100)

ORDER BY

# Order by requires that sample is numeric and has 1 value,
# i.e. no lists or multi-dimensional arrays
select * where contains(tensor_name, 'text_value') order by tensor_name asc

ANY, ALL, and ALL_STRICT

select * where all_strict(tensor_name[:,2]>numeric_value)
select * where any(tensor_name[0:6]>numeric_value)
all adheres to NumPy and list logic where all(empty_sample) returns True
all_strict is more intuitive for queries so all_strict(empty_sample) returns False
LOGICAL_AND and LOGICAL_OR
select * where any(logical_and(tensor_name_1[:,3]>numeric_value, tensor_name_2 == 'text_value'))
Copy link
On this page
How to query machine learning datasets using Activeloop's query engine
Dataset Query Summary
Dataset Query Syntax