v2.0.6
EXAMPLE CODE
DOCUMENTATION
How to Contribute
Let's make Hub 2.0 amazing with your help! Any contribution is appreciated, whether it's code profiling, refactoring, and tests.
Contributing Guidelines
Linting
Hub uses the black python linter. You can have your code auto-formatted by running
pip install black, then black . inside the directory you want to format.Docstrings
Typing
Hub uses static typing for function arguments/variables for better code readability. Hub has a GitHub action that runs
mypy ., which runs similar to pytest . to check for valid static typing. You can refer to mypy documentation for more information.Testing
Hub uses pytest for tests. In order to make it easier to contribute, Hub also has a set of custom options defined in conftest.py.
Prerequisites
Options
To see a list of Hubs custom pytest options, run this command:
pytest -h | sed -En '/custom options:/,/\[pytest\] ini\-options/p'.Fixtures
memory_storage: If--memory-skipis provided, tests with this fixture will be skipped. Otherwise, the test will run with only aMemoryProvider.local_storage: If--localis not provided, tests with this fixture will be skipped. Otherwise, the test will run with only aLocalProvider.s3_storage: If--s3is not provided, tests with this fixture will be skipped. Otherwise, the test will run with only anS3Provider.storage: All tests that use thestoragefixture will be parametrized with the enabledStorageProviders (enabled via options defined below). If--cache-chainsis provided,storagemay also be a cache chain. Cache chains have the same interface asStorageProvider, but instead of just a single provider, it is multiple chained in a sequence, where the last provider in the chain is considered the actual storage.ds: The same as thestoragefixture, but the storages that are parametrized are wrapped with aDataset.
Each
StorageProvider/Dataset that is created for a test via a fixture will automatically have a root created before and destroyed after the test. If you want to keep this data after the test run, you can use the --keep-storage option.Fixture Examples
Single storage provider fixture
1
def test_memory(memory_storage):
2
# Test will skip if `--memory-skip` is provided
3
memory_storage["key"] = b"1234" # This data will only be stored in memory
4
5
def test_local(local_storage):
6
# Test will skip if `--local` is not provided
7
memory_storage["key"] = b"1234" # This data will only be stored locally
8
9
def test_local(s3_storage):
10
# Test will skip if `--s3` is not provided
11
# Test will fail if credentials are not provided
12
memory_storage["key"] = b"1234" # This data will only be stored in s3
Copied!
Multiple storage providers/cache chains
1
from hub.core.tests.common import parametrize_all_storages, parametrize_all_caches, parametrize_all_storages_and_caches
2
3
@parametrize_all_storages
4
def test_storage(storage):
5
# Storage will be parametrized with all enabled `StorageProvider`s
6
pass
7
8
@parametrize_all_caches
9
def test_caches(storage):
10
# Storage will be parametrized with all common caches containing enabled `StorageProvider`s
11
pass
12
13
@parametrize_all_storages_and_caches
14
def test_storages_and_caches(storage):
15
# Storage will be parametrized with all enabled `StorageProvider`s and common caches containing enabled `StorageProvider`s
16
pass
Copied!
Dataset storage providers/cache chains
1
from hub.core.tests.common import parametrize_all_dataset_storages, parametrize_all_dataset_storages_and_caches
2
3
@parametrize_all_dataset_storages
4
def test_dataset(ds):
5
# `ds` will be parametrized with 1 `Dataset` object per enabled `StorageProvider`
6
pass
7
8
@parametrize_all_dataset_storages_and_caches
9
def test_dataset(ds):
10
# `ds` will be parametrized with 1 `Dataset` object per enabled `StorageProvider` and all cache chains containing enabled `StorageProvider`s
11
pass
Copied!
Benchmarks
TODO: benchmarking is subject to change. will update this section once it is better defined.
Thank you!
Hub would not be possible without the work of our community.
Last modified 10mo ago