Storage Synchronization
Synchronizing data with long-term storage and achieving optimal performance using Hub.
Using
with context when updating hub datasets is critical for achieving rapid write performance.BAD PRACTICE - Code without with context
with contextBy default, any standalone update to a hub dataset is immediately pushed to the dataset's long-term storage location. Due to the sheer number of discreet write operations, there may be a significant increase in runtime, especially when the data is stored in the cloud. In the example below, an update is pushed to storage for every call to the
.append() command.1
for i in range(10):
2
ds.my_tensor.append(i)
Copied!
Code using with context
with contextTo reduce the runtime when using Hub, the
with syntax below significantly improves performance because it only pushes updates to long-term storage after the code block inside the with statement has been executed, or when the local cache is full. This significantly reduces the number of discreet write operations, thereby increasing the speed by up to 100X. 1
with ds:
2
for i in range(10):
3
ds.my_tensor.append(i)
Copied!
Last modified 27d ago
Copy link