Updating Datasets
Updating Deep Lake datasets
How to make updates to Deep Lake datasets
This tutorial is also available as a Colab Notebook
After creating a Deep Lake dataset, you may need to edit it by adding, deleting, and modifying the data. In this tutorial, we show best practices for updating datasets.
Create a Representative Deep Lake Dataset
First, let's download and unzip representative source data and create a Deep Lake dataset for this tutorial:
This dataset includes segmentation and object detection of vehicle damage, but for this tutorial, we will only upload the images and labels (damage location)
ds.summary()
shows the dataset has two tensors with 11 samples:
We can explore the damage in the first sample using ds.labels[0].data()
, which prints:
Add Data to a New Tensor
Suppose you have another data source with supplemental data about the color of the vehicles. Let's create a Pandas DataFrame with this data.
There are two approaches for adding this new data to the Deep Lake dataset:
1. Iterate through the Deep Lake samples and append data
This approach is recommended when most Deep Lake samples are being updated using the supplemental data (dense update).
First, we create a color
tensor and iterate through the samples. For each sample, we lookup the color from the df_color
DataFrame and append it to the color
tensor. If no color exists for a filename, it is appended as None
. We use the filename as the key to perform the lookup, which is available in ds.images[index].sample_info
dictionary.
Now we see that ds.summary()
shows 3 tensors, each with 11 samples (though the color
tensor has several empty samples):
Iterate through the supplemental data and add data at the corresponding Deep Lake index
This approach is recommended when the data updates are sparse
First, let's create a color2
tensor, and the load all the existing Deep Lake filenames into memory. We then iterate through the supplemental data and find the corresponding Deep Lake index to insert the color information.
Now we see that ds.summary()
shows 4 tensors, each with 11 samples (though the color
and color2
tensors have several empty samples):
Update Existing Rows without TQL
Originally, we did not specify a color for image 3.jpg
. Let's find the index for this image, look at it, and add the color manually. We've already loaded the Deep Lake dataset's filenames into memory above, so we can find the index using:
Let's visualize the image using PIL. We could also visualize it using ds.visualize()
(must pip install "deeplake[visualizer]"
) or using the Deep Lake App.
Since the image is white, let's update the color using:
Delete Samples
Rows from a dataset can be deleted using ds.pop()
. To delete the row at index 8 we run:
Now we see that ds.summary()
shows 10 rows in the dataset (instead of 11):
To replace data with empty data without deleting a row, you can run:
Congrats! You just learned how to make a variety of updates to Deep Lake datasets! 🎉