Step 6: Connecting Hub Datasets to ML Frameworks

GitHub Activeloop Login Slack Community Short Survey

Search…

Connecting Hub Datasets to machine learning frameworks such as PyTorch and TensorFlow.

You can connect Hub Datasets to popular ML frameworks such as PyTorch and TensorFlow using minimal boilerplate code, and Hub takes care of the parallel processing!
PyTorch
You can train a model by creating a PyTorch DataLoader from a Hub Dataset using ds.pytorch() .
1
import hub
2
from torch.utils.data import DataLoader
3
​
4
ds = hub.dataset('hub://activeloop/cifar100-train') # Hub Dataset
5
dataloader= ds.pytorch(batch_size = 16, num_workers = 2) #PyTorch Dataloader
6
​
7
for data in dataloader:
8
    print(data)
9
    # Training Loop
Copied!
TensorFlow
Similarly, you can convert a Hub Dataset to a TensorFlow Dataset via the tf.Data API. 
1
ds # Hub Dataset object, to be used for training
2
ds_tf = ds.tensorflow() # A TensorFlow Dataset
Copied!

Step 5: Using Activeloop Storage

Step 7: Parallel Computing

Last modified 8d ago

Copy link

Contents

PyTorch

TensorFlow