Comment on page
Tensor Query Language (TQL)
Deep Lake offers a highly-performant SQL-style query engine for filtering your data.
Querying datasets is a critical aspect of data science workflows that enables users to filter datasets and focus their work on the most relevant data. Deep Lake offers a highly-performant query engine built in C++ and optimized for the Deep Lake data format.
The Deep Lake query engine is only accessible to registered and authenticated users, and it applies usage restrictions based on your Deep Lake Plan.
view = vector_store.search(query = <query_string>, exec_option = "compute_engine")
Queries can also be performed in the Python API using:
view = ds.query(<query_string>)
ds_view.save_view(message = 'Samples with monarchs')
In order to maintain data lineage,
Dataset Viewsare immutable and are connected to specific commits. Therefore, views can only be saved if the dataset has a commit and there are no uncommitted changes in the
HEAD. You can check for this using
Dataset Viewscan be loaded in the python API and they can passed to ML frameworks just like regular datasets:
ds_view = ds.load_view(view_id, optimize = True, num_workers = 2)
for data in ds_view.pytorch():
# Training loop here
optimize = True
Dataset Viewinto a new sub-dataset that is optimized for streaming. If the original dataset uses linked tensors, the data will be copied to Deep Lake format.
Dataset Viewis critical for achieving rapid streaming.
If the saved
Dataset Viewis no longer needed, it can be deleted using: