Querying Datasets
Activeloop Platform offer a highly-performant SQL-style query engine for filtering your data.
How to query machine learning datasets using Activeloop's query engine
Querying datasets is a critical aspect of data science workflows that enables users to filter datasets and focus their work on the most relevant data at hand. Activeloop offers a highly-performant dataset query engine built in C++ and optimized for Deep Lake datasets.
Starting from deeplake 3.1.0
, querying your own datasets will require the dataset's parent organization to be on a Growth of Enterprise plan. Querying Activeloop datasets will be enabled for everyone.
Dataset Query Summary
Querying in the UI
Querying in the Python API
Queries can also be performed in the Python API using:
Saving and utilizing dataset query results
The query results (Dataset Views
) can be saved in the UI as shown above, or if the view is generated in Python, it can be saved using the Python API below. Full details are available here.
In order to maintain data lineage, Dataset Views
are immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the HEAD
. You can check for this using ds.has_head_changes
Dataset Views
can be loaded in the python API and they can passed to ML frameworks just like regular datasets:
The optimize
parameter in ds.load_view(...,
optimize = True
)
materializes the Dataset View
into a new sub-dataset that is optimized for streaming. If the original dataset uses linked tensors, the data will be copied to Deep Lake format.
Optimizing the Dataset View
is critical for achieving rapid streaming.
If the saved Dataset View
is no longer needed, it can be deleted using:
Dataset Query Syntax
CONTAINS and ==
Any special characters in tensor or group names should be wrapped with double-quotes:
SHAPE
LIMIT
AND, OR, NOT
UNION and INTERSECT
ORDER BY
ANY, ALL, and ALL_STRICT
all
adheres to NumPy and list logic where all(empty_sample)
returns True
all_strict
is more intuitive for queries so all_strict(empty_sample)
returns False
LOGICAL_AND and LOGICAL_OR
SAMPLE BY
weight_choice
resolves the weight that is used when multiple expressions evaluate toTrue
for a given sample. Options aremax_weight, sum_weight
. For example, ifweight_choice
ismax_weight
, then the maximum weight will be chosen for that sample.replace
determines whether samples should be drawn with replacement. It defaults toTrue
.limit
specifies the number of samples that should be returned. If unspecified, the sampler will return the number of samples corresponding to the length of the dataset
Last updated