How to Contribute
Guidelines and tips for contributing to our open-source project.

Hub relies on feedback and contributions from our wonderful community. Let's make it amazing with your help! Any and all contributions are appreciated, including code profiling, refactoring, and tests.

Providing Feedback

Getting Started With Development

Clone the repository:
1
git clone https://github.com/activeloopai/hub
2
cd Hub
Copied!
If you are using Linux, install environment dependencies:
1
apt-get -y update
2
apt-get -y install git wget build-essential python-setuptools python3-dev libjpeg-dev libpng-dev zlib1g-dev
3
apt install build-essential
Copied!
If you are planning to work on videos, install codecs:
1
apt-get install -y ffmpeg libavcodec-dev libavformat-dev libswscale-dev
Copied!
Install the package locally with plugins and development dependencies:
1
pip install -r hub/requirements/plugins.txt
2
pip install -r hub/requirements/tests.txt
3
pip install -e .
Copied!
Run local tests to ensure everything is correct:
1
pytest -x --local .
Copied!

Using Docker (optional)

You can use docker-compose for running tests
1
docker-compose -f ./bin/docker-compose.yaml up --build local
Copied!
and even work inside the docker by building the image and bashing into.
1
docker build -t activeloop-hub:latest -f ./bin/Dockerfile.dev .
2
docker run -it -v $(pwd):/app activeloop-hub:latest bash
3
$ python3 -c "import hub"
Copied!
Now changes done on your local files will be directly reflected into the package running inside the docker.

Contributing Guidelines

Linting

Hub uses the black python linter. You can auto-format your code by running pip install black, and the run black . inside the directory you want to format.

Docstrings

Hub uses Google Docstrings. Please refer to this example to learn more.

Typing

Hub uses static typing for function arguments/variables for better code readability. Hub has a GitHub action that runs mypy ., which runs similar to pytest . to check for valid static typing. You can refer to mypy documentation for more information.

Testing

Hub uses pytest for tests. In order to make it easier to contribute, Hub also has a set of custom options defined here.

Prerequisites

Options

To see a list of Hub's custom pytest options, run this command: pytest -h | sed -En '/custom options:/,/\[pytest\] ini\-options/p'.

Fixtures

You can find more information on pytest fixtures here.
  • memory_storage: If --memory-skip is provided, tests with this fixture will be skipped. Otherwise, the test will run with only a MemoryProvider.
  • local_storage: If --local is not provided, tests with this fixture will be skipped. Otherwise, the test will run with only a LocalProvider.
  • s3_storage: If --s3 is not provided, tests with this fixture will be skipped. Otherwise, the test will run with only an S3Provider.
  • storage: All tests that use the storage fixture will be parametrized with the enabled StorageProviders (enabled via options defined below). If --cache-chains is provided, storage may also be a cache chain. Cache chains have the same interface as StorageProvider, but instead of just a single provider, it is multiple chained in a sequence, where the last provider in the chain is considered the actual storage.
  • ds: The same as the storage fixture, but the storages that are parametrized are wrapped with a Dataset.
Each StorageProvider/Dataset that is created for a test via a fixture will automatically have a root created, and it will be destroyed after the test. If you want to keep this data after the test run, you can use the --keep-storage option.

Fixture Examples

Single storage provider fixture:
1
def test_memory(memory_storage):
2
# Test will skip if `--memory-skip` is provided
3
memory_storage["key"] = b"1234" # This data will only be stored in memory
4
5
def test_local(local_storage):
6
# Test will skip if `--local` is not provided
7
memory_storage["key"] = b"1234" # This data will only be stored locally
8
9
def test_local(s3_storage):
10
# Test will skip if `--s3` is not provided
11
# Test will fail if credentials are not provided
12
memory_storage["key"] = b"1234" # This data will only be stored in s3
Copied!
Multiple storage providers/cache chains:
1
from hub.core.tests.common import parametrize_all_storages, parametrize_all_caches, parametrize_all_storages_and_caches
2
3
@parametrize_all_storages
4
def test_storage(storage):
5
# Storage will be parametrized with all enabled `StorageProvider`s
6
pass
7
8
@parametrize_all_caches
9
def test_caches(storage):
10
# Storage will be parametrized with all common caches containing enabled `StorageProvider`s
11
pass
12
13
@parametrize_all_storages_and_caches
14
def test_storages_and_caches(storage):
15
# Storage will be parametrized with all enabled `StorageProvider`s and common caches containing enabled `StorageProvider`s
16
pass
Copied!
Dataset storage providers/cache chains:
1
from hub.core.tests.common import parametrize_all_dataset_storages, parametrize_all_dataset_storages_and_caches
2
3
@parametrize_all_dataset_storages
4
def test_dataset(ds):
5
# `ds` will be parametrized with 1 `Dataset` object per enabled `StorageProvider`
6
pass
7
8
@parametrize_all_dataset_storages_and_caches
9
def test_dataset(ds):
10
# `ds` will be parametrized with 1 `Dataset` object per enabled `StorageProvider` and all cache chains containing enabled `StorageProvider`s
11
pass
Copied!

Benchmarks

Hub uses pytest-benchmark for benchmarking, which is a plugin for pytest.

Thank you!

Hub would not be possible without the work of our community.