Step 4: Accessing Data

GitHub Activeloop Login Slack Community Short Survey

Search…

Accessing and loading Hub Datasets.

Loading Datasets
Hub Datasets can be loaded and created in a variety of storage locations with minimal configuration. 
1
import hub
2
​
3
# Local Filepath
4
ds = hub.load('./my_dataset_path')
5
​
6
# S3
7
ds = hub.load('s3://my_dataset_bucket', creds={...})
8
​
9
## Activeloop Storage - See Step 6
10
# Public Dataset hosted by Activeloop
11
ds = hub.load('hub://activeloop/public_dataset_name')
12
​
13
# Dataset in another workspace on Activeloop Platform
14
ds = hub.load('hub://workspace_name/dataset_name')
Copied!
Since ds = hub.dataset(path)can be used to both create and load datasets, you may accidentally create a new dataset if there is a typo in the path you provided while intending to load a dataset. If that occurs, simply use ds.delete() to remove the unintended dataset permanently.
Referencing Tensors
Hub allows you to reference specific tensors using keys or via the "." notation outlined below. 
Note: data is still not loaded by these commands.
1
### NO HIERARCHY ###
2
ds.images # is equivalent to
3
ds['images']
4
​
5
ds.labels # is equivalent to
6
ds['labels']
7
​
8
### WITH HIERARCHY ###
9
ds.localization.boxes # is equivalent to
10
ds['localization/boxes']
11
​
12
ds.localization.labels # is equivalent to
13
ds['localization/labels']
Copied!
Accessing Data
Data within the tensors is loaded and accessed using the .numpy() command:
1
# Indexing
2
W = ds.images[0].numpy() # Fetch an image and return a NumPy array
3
X = ds.labels[0].numpy(aslist=True) # Fetch a label and store it as a 
4
                                    # list of NumPy arrays
5
​
6
# Slicing
7
Y = ds.images[0:100].numpy() # Fetch 100 images and return a NumPy array
8
                             # The method above produces an exception if 
9
                             # the images are not all the same size
10
​
11
Z = ds.labels[0:100].numpy(aslist=True) # Fetch 100 labels and store 
12
                                         # them as a list of NumPy arrays
Copied!
The .numpy()method will produce an exception if all samples in the requested tensor do not have a uniform shape. If that's the case, running .numpy(aslist=True)solves the problem by returning a list of NumPy arrays, where the indices of the list correspond to different samples. 
​

Step 3: Understanding Compression

Step 5: Using Activeloop Storage

Last modified 8d ago

Copy link

Contents

Loading Datasets

Referencing Tensors

Accessing Data