sample
or chunk
level. For example, when creating a tensor for storing images, you can choose the compression technique for the image samples using the sample_compression
input:.append(...)
calls is compressed using the specified sample_compression
method. image
, audio
, video
,...), choose the sample_compression
technique that is native to the application such as jpg
, mp3
, mp4
,...lz4
is a generic compressor that works well in most applications.lz4
can be used as a sample_compression
or chunk_compression
. In most cases, sample_compression
is sufficient, but in theory, chunk_compression
produces slightly smaller data.png
vs jpeg
compression may result in a 10X difference in the size of a Hub dataset. sample_compression
input matches the compression of the source data, such as:.jpg
, and the hub sample_compression
is jpg
. sample_compression
in Hub results in significantly slower upload speeds, because Hub must decompress the source data and recompress it using the specified sample_compression
before saving.sample_compression
.