Active learning as a framework has a straight objective to select the data points, or images, that would be most beneficial for the deep learning model to learn from. As a cyclic framework it consists of Training pool, a machine learning, a strategy to estimate uncertainty of unlabeled data points, unlabeled pool of data and an Oracle that annotates the data. The cyclic framework is in short a data engine that is designed to keep improving the machine learning model [1].

In MONAI label, the developer creates their own scoring strategy based on which the next sample can be chosen. All sample-apps are equipped with a basic skeleton to allow for the development of a scoring strategy. For example here in the DeepEdit App.

Currently, MONAI Label offers three Active Learning strategies that researchers can use to accelerate the training process. Two are based on the uncertainty that comes from the model (Epistemic Uncertainty or model-based uncertainty as referred in scientific literature) using Dropout namely Entropy & Variance. The other one is based on the uncertainty that comes from the dataset (Aleatoric uncertainty or data-based uncertainty) using Test Time Augmentation (TTA).

Epistemic Uncertainty or model-based uncertainty is a scoring strategy where data samples are scored based on a measure of the uncertainty as per the model. It can be observed in the below figure that certain connection in a neural network are dropped during inference to generate varying predictions. The varying predictions can be observed in the focus boxes in the below figure. Variance is computed given 'N' predictions, this variance is used as uncertainty to rank unlabeled data to select 'most uncertain' samples for annotation

It can be observed in the figure below that if dropout is introduced in different layers of the network, it can be observed that the predictions are different every time different connections between the layers are dropped out. The entropy measure across the multiple predictions is termed as the uncertainty of the image

Test time augmentations [2] are a useful feature for computing uncertainty, as well as observing the network's dependency on the applied random transforms. MONAI Label offers TTA as one of the scoring methods. TTA uses transform(s) that are applied to each image. Those transforms should have their corresponding inverse so the image can be returned to the same spatial reference. Then metrics such as the volume variation coefficient (vvc) can be computed and used as a scoring method.

Reference:

[1] Nath, V., Yang, D., Landman, B. A., Xu, D., & Roth, H. R. (2020). Diminishing uncertainty within the training pool: Active learning for medical image segmentation. IEEE Transactions on Medical Imaging, 40(10), 2534-2547.

[2] Wang et al., Aleatoric uncertainty estimation with test-time augmentation for medical image segmentation with convolutional neural networks, https://doi.org/10.1016/j.neucom.2019.01.103