Working with Videos
How manage video datasets and train models using Deep Lake.
Performing deep-learning on video data can be challenging due to the large size of video files, especially when they are uncompressed to raw numeric data that is fed into neural networks. Deep Lake abstracts these challenges away from the user so you can focus on building performant models.
Make sure to install Deep Lake with
pip install "deeplake[av]" in order to use Deep Lake's audio and video features.import deeplake
ds = deeplake.empty("demo/video") # create a local dataset
To create a video tensor, we specify an
htype of "video" and set sample_compression to the format of the video.ds.create_tensor("videos", htype="video", sample_compression="mp4")
We append videos to the newly created tensor by reading the video files with
deeplake.readds.videos.append(deeplake.read("./videos/example1.mp4"))
ds.videos.append(deeplake.read("./videos/example2.mp4"))
deeplake.read can also read videos from http://, gcs:// and s3:// urls given you have the credentials to access them. Examples include:ds.videos.append(
deeplake.read(
"http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4",
creds=None,
)
)
ds.videos.append(
deeplake.read(
"s3://bucket-name/sample_video.mp4",
creds={
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_session_token": "...",
},
)
)
Annotations like bounding boxes can be added and visualized in Deep Lake along with the video samples. We use tensors of htype
sequence[bbox] for this purpose. Every sample in a sequence[bbox] tensor will be a sequence of bounding boxes which represents the annotations for the corresponding video sample in the video tensor.ds.create_tensor("boxes", htype="sequence[bbox]", coords={"type": "pixel", "mode": "LTWH"})
Next, here's an example of an annotations file taken from the LaSOT dataset. It contains annotations for every frame of a video.
groundtruth.txt
39KB
Text
Example of an annotations file
We convert this to a numpy array and append it to our
boxes tensor.import pandas as pd
df = pd.read_csv("groundtruth.txt", header=None)
boxes = df.to_numpy().astype(np.float32)[:, np.newaxis]
# boxes.shape == (2788, 1, 4) == (number of frames, number of boxes, 4)
ds.boxes.append(boxes)
Visualize the bounding boxes within your notebook using
ds.visualize().The shapes of the samples in the
video and sequence[bbox]tensors have to match in order for visualization to work properly.If the shape of video tensor is
(# frames, height, width, 3), the shape of the sequence tensor should be(# frames, # of boxes in a frame, 4)
Visualize video annotations in notebook
We can get the shape of a video sample in (N, H, W, C) format using
ds.videos[0].shape
(400, 360, 640, 3)
Info about a video sample can be accessed using
ds.videos[0].sample_info
This returns info about the first sample as a dict:
{
'duration': 400400,
'fps': 29.97002997002997,
'timebase': 3.3333333333333335e-05,
'shape': [400, 360, 640, 3],
'format': 'mp4',
'filename': './videos/example1.mp4',
'modified': False
}
duration is in units oftimebaseThe most important part of working with videos on Deep Lake is retrieving the frames of a video sample as a numpy array.
video = ds.videos[0].numpy()
This decompresses the entire first video sample and returns the frames as a numpy array.
print(type(video))
print(video.shape)
<class 'numpy.ndarray'>
(400, 360, 640, 3)
Be careful when decompressing an entire large video sample because it can blow up your memory.
Deep Lake allows you to index the video tensor like a numpy array and return the frames you want. Only the required frames are decompressed. See a few examples below:
Getting a 100 frames from index 100 - 200
# 1st sample, frames 100 - 200
video = ds.videos[1, 100:200].numpy()
video.shape
(100, 360, 640, 3)
Indexing with step
# 0th sample, frames 100 - 200 with step of 5 frames
video = ds.videos[0, 100:200:5].numpy()
video.shape
(20, 360, 640, 3)
Getting a single frame
# 1st sample, last frame
last_frame = ds.videos[1, -1].numpy()
last_frame.shape
(360, 640, 3)
Presentation timestamps (PTS) of frames can be obtained (in seconds) through a video tensor's
.timestamp attribute after indexing it just like in the previous section:# timestamps of frames 10 - 15 of 0th sample
ds.videos[0, 10:15].timestamp
array([0.36703333, 0.4004 , 0.43376666, 0.46713334, 0.5005 ],
dtype=float32)
Calling
ds.videos[index].data() will return a dict with keys 'frames' and 'timestamps' with the corresponding numpy arrays as values. Indexing works the same way as it does with .numpy().data = ds.videos[1, 15:20].data()
data['frames'].shape
data['timestamps']
(5, 360, 640, 3)
array([0.5005 , 0.5672333 , 0.6006 , 0.6339667 , 0.76743335],
dtype=float32)
Individual video samples can be instantly visualized by calling
.play() on them:ds.videos[1].play()
This will play the video on your web browser:
video playback in browser
On a jupyter notebook this will look like:
video playback on jupyter notebook
This feature is not yet supported on colab
The whole Deep Lake dataset can be visualized by calling
.visualize() on your dataset in a jupyter or colab notebook.ds.visualize()
ds.visualize() on colab
On colab, we only support visualizing
hub://datasetsTensors of Deep Lake type
link[video] can be used to store links to videos. All of the above features are supported for linked videos. https://, gcs://, s3://and gdrive://links are accepted.# create linked tensor
links = ds.create_tensor("video_links", htype="link[video]")
# append linked samples
links.append(deeplake.link("http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/ForBiggerBlazes.mp4", creds_key=None)) # link to public video
# .numpy()
video = links[0].numpy()
# shape of numpy array
video.shape
(360, 720, 1280, 3)
You will need to set credentials to link to private data on your S3 or GCS.
This proccess is easy and streamlined for
deeplake:// datasets.- And then choose 'Add Credentials'.
- Select a credentials provider, set the credentials name (say, 'MY_KEY'), fill the fields and save it.
- Done! Your credentials have now been set.
Use
ds.add_creds_key with managed set to True to add the credentials to your dataset. Multiple credentials can be added.ds.add_creds_key("MY_KEY", managed=True)
ds.add_creds_key("S3_KEY", managed=True)
And when adding linked data using
deeplake.link, simply mention which credentials to use through the creds_key argument.ds.links.append(deeplake.link("s3://my-bucket/sample-video.mp4", creds_key="MY_KEY"))
For non-
hub:// datasets, you can use credentials set in your environment by mentioning creds_key="ENV"ds.links.append(deeplake.link("s3://my-bucket/sample-video.mp4", creds_key="ENV"))
Or you can temporarily add creds to your dataset
creds={
"aws_access_key_id": "...",
"aws_secret_access_key": "...",
"aws_session_token": "...",
}
# add creds key (Note that managed is False)
ds.add_creds_key("TEMP_KEY")
# populate creds with a credentials dict
ds.populate_creds("TEMP_KEY", creds)
and then
ds.links.append(deeplake.link("s3://my-bucket/sample-video.mp4", creds_key="TEMP_KEY"))
This section describes some implementation details regarding how video data is fetched and decompressed in Deep Lake.
Large video samples (> 16MB by default) stored in remote Deep Lake datasets are not downloaded in their entirety on calling
.numpy(). Instead, they are streamed from storage. Only the required packets are decompressed and converted to numpy arrays based on how the tensor is indexed..play()also streams videos from storage.