# Querying Datasets

Deep Lake offers a highly-performant SQL-style query engine for filtering your data.

Querying datasets is a critical aspect of data science workflows that enables users to filter datasets and focus their work on the most relevant data at hand. Activeloop offers a highly-performant dataset query engine built in C++ and optimized for Deep Lake datasets.

Querying features in the python API are installed using

`pip install "deeplake[enterprise]"`

. Details on all installation options are available here. Queries can also be performed in the Python API using:

view = ds.query('Query string')

The query results (

`Dataset Views`

) can be saved in the UI as shown above, or if the view is generated in Python, it can be saved using the Python API below. Full details are available here.ds_view.save_view(message = 'Samples with monarchs')

In order to maintain data lineage,

`Dataset Views`

are immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the `HEAD`

. You can check for this using `ds.has_head_changes`

`Dataset Views`

can be loaded in the python API and they can passed to ML frameworks just like regular datasets:ds_view = ds.load_view(view_id, optimize = True, num_workers = 2)

for data in ds_view.pytorch():

# Training loop here

The

`optimize `

parameter in `ds.load_view(..., `

`optimize = True`

`)`

materializes the `Dataset View`

into a new sub-dataset that is optimized for streaming. If the original dataset uses linked tensors, the data will be copied to Deep Lake format.Optimizing the

`Dataset View`

is critical for achieving rapid streaming.If the saved

`Dataset View`

is no longer needed, it can be deleted using:ds.delete_view(view_id)

# Exact match, which generally requires that the sample

# has 1 value, i.e. no lists or multi-dimensional arrays

select * where tensor_name == 'text_value' # If value is numeric

select * where tensor_name == numeric_value # If values is text

select * where contains(tensor_name, 'text_value')

Any special characters in tensor or group names should be wrapped with double-quotes:

select * where contains("tensor-name", 'text_value')

select * where "tensor_name/group_name" == numeric_value

select * where shape(tensor_name)[dimension_index] > numeric_value

select * where shape(tensor_name)[1] > numeric_value # Second array dimension > value

select * where contains(tensor_name, 'text_value') limit num_samples

select * where contains(tensor_name, 'text_value') and NOT contains(tensor_name_2, numeric_value)

select * where contains(tensor_name, 'text_value') or tensor_name_2 == numeric_value

select * where (contains(tensor_name, 'text_value') and shape(tensor_name_2)[dimension_index]>numeric_value) or contains(tensor_name, 'text_value_2')

(select * where contains(tensor_name, 'value')) intersect (select * where contains(tensor_name, 'value_2'))

(select * where contains(tensor_name, 'value') limit 100) union (select * where shape(tensor_name)[0] > numeric_value limit 100)

# Order by requires that sample is numeric and has 1 value,

# i.e. no lists or multi-dimensional arrays

select * where contains(tensor_name, 'text_value') order by tensor_name asc

select * where all_strict(tensor_name[:,2]>numeric_value)

select * where any(tensor_name[0:6]>numeric_value)

**adheres to NumPy and list logic where**

`all`

`all(empty_sample)`

returns `True`

**is more intuitive for queries so**

`all_strict`

`all_strict(empty_sample)`

returns `False`

**LOGICAL_AND**and

**LOGICAL_OR**

select * where any(logical_and(tensor_name_1[:,3]>numeric_value, tensor_name_2 == 'text_value'))

select * sample by weight_choice(expression_1: weight_1, expression_2: weight_2, ...)

replace True limit N

resolves the weight that is used when multiple expressions evaluate to`weight_choice`

`True`

for a given sample. Options are`max_weight, sum_weight`

. For example, if`weight_choice`

is`max_weight`

, then the maximum weight will be chosen for that sample.

determines whether samples should be drawn with replacement. It defaults to**replace**`True`

.specifies the number of samples that should be returned. If unspecified, the sampler will return the number of samples corresponding to the length of the dataset`limit`

Last modified 1mo ago