Sampling Datasets
Implementation of samplers in TQL
How to sample datasets using Deep Lake's query engine
Sampling is often used when training models in order to modify the distribution of data that models are trained on. A common objective of samplers is to rebalance the data in order to achieve an more uniform distribution of classes in the training loop. Deep Lake provides a powerful API for several sampling methods via the query engine.
The general syntax for sampling is using the sample by
keywords:
weight_choice
resolves the weight that is used when multiple expressions evaluate toTrue
for a given sample. Options aremax_weight, sum_weight
. For example, ifweight_choice
ismax_weight
, then the maximum weight will be chosen for that sample.replace
determines whether samples should be drawn with replacement. It defaults toTrue
.limit
specifies the number of samples that should be returned. If unspecified, the sampler will return the number of samples corresponding to the length of the dataset
Sampling can be performed in the query interface in the Deep Lake UI, or in the Python API as shown below.
Example Usage
Next, let's calculate the a histogram of the medical findings (findings
tensor) and plot it.
We observe that findings such as Hernia
, Pneumonia
, Fibrosis
, Edema
, and Emphysema
are very rare, which may cause our model to underperform when predicting these conditions. Note that even though many images have No_Finding
, this is desirable for avoiding false positives when training models for medical imaging applications.
We can use Deep Lake Tensor-Query-Language to upsample the under-represented findings in order to create a more balanced dataset.
We can run this query in the UI or in the Python API using ds.query(...)
:
In this sampler query, we're upsampling Hernia
, by 20x, Pneumonia
by 8x, Fibrosis
by 5x, Edema
by 5x, and Emphysema
by 2x. Let's recalculate the histogram for the balanced dataset and compare it to the raw data histogram.
The data in the upsampled dataset has much better representation of the rare conditions. Note that since a given image may have multiple conditions, and since conditions can be correlated, upsampling by one condition may implicitly upsample another condition, if they tend to occur in the same image.
Training Models on Sampled Views
Last updated