Storage Options
How to authenticate using Activeloop storage, AWS S3, and Google Cloud Storage.
Deep Lake datasets can be stored locally, or on several cloud storage providers including Deep Lake Storage, AWS S3, Microsoft Azure, and Google Cloud Storage. Datasets are accessed by choosing the correct prefix for the dataset path
that is passed to methods such as deeplake.load(path)
, and deeplake.empty(path)
. The path prefixes are:
Connecting Deep Lake datasets stored in your own cloud via Deep Lake Managed Credentials is required for accessing enterprise features, and it significantly simplifies dataset access.
Authentication for each cloud storage provider:
Activeloop Storage and Managed Datasets
In order to access datasets stored in Deep Lake, or datasets in other clouds that are managed by Activeloop, users must register and authenticate using the steps in the link below:
AWS S3
Authentication with AWS S3 has 4 options:
Use Deep Lake on a machine in the AWS ecosystem that has access to the relevant S3 bucket via AWS IAM, in which case there is no need to pass credentials in order to access datasets in that bucket.
Configure AWS through the cli using
aws configure
. This creates a credentials file on your machine that is automatically access by Deep Lake during authentication.Save the
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
, andAWS_SESSION_TOKEN (optional)
in environmental variables of the same name, which are loaded as default credentials if no other credentials are specified.Create a dictionary with the
AWS_ACCESS_KEY_ID
,AWS_SECRET_ACCESS_KEY
, andAWS_SESSION_TOKEN (optional)
, and pass it to Deep Lake using:Note: the dictionary keys must be lowercase!
endpoint_url
can be used for connecting to other object storages supporting S3-like API such as MinIO, StorageGrid and others.
Custom Storage with S3 API
In order to connect to other object storages supporting S3-like API such as MinIO, StorageGrid and others, simply add endpoint_url
the the creds
dictionary.
Microsoft Azure
Authentication with Microsoft Azure has 4 options:
Log in from your machine's CLI using
az login
.Save the
AZURE_STORAGE_ACCOUNT
,AZURE_STORAGE_KEY
, or other credentials in environmental variables of the same name, which are loaded as default credentials if no other credentials are specified.Create a dictionary with the
ACCOUNT_KEY
orSAS_TOKEN
and pass it to Deep Lake using:Note: the dictionary keys must be lowercase!
Google Cloud Storage
Authentication with Google Cloud Storage has 2 options:
Create a service account, download the JSON file containing the keys, and then pass that file to the
creds
parameter indeeplake.load('gcs://.....', creds = 'path_to_keys.json')
. It is also possible to manually pass the information from the JSON file into thecreds
parameter using:Authenticate through the browser using the steps below. This requires that the project credentials are stored on your machine, which happens after
gcloud
is initialized and logged in through the CLI. Afterwards,creds
can be switched tocreds = 'cache'
.