A common issue present with AI in medical imaging is the lack of ample, curated data for training predictive models, as annotation generation is often prohibitively resource-intensive. We propose the implementation of a continuous AI-assisted feedback loop to accelerate image annotation generation, in which a starter AI model, trained on limited data, provides annotation initializations to human annotators to accelerate annotation, and the subsequent human-corrected annotations are sent back to update the AI model. This loop is run continuously with small batches of data, with each iteration improving the AI-initialized annotation, and thereby requiring even less time for human annotation. Eventually, minimal annotation is needed, requiring only a fraction of resources to generate well-annotated data. We demonstrate the utility of this approach, even with small batches (n < 10), in an initial implementation for kidney segmentation in CT images.*Methods and Materials An off-the-shelf 3D U-Net was trained on an initial set of 8 CT images with kidney annotations from an expert image analyst. This initial model was used to generate initial segmentations on unannotated images, which were sent back the analyst to aid them in annotating the kidneys. The newly annotated images were added to the training data for subsequent updating of the model. This feedback loop was repeated for 2 iterations, for a total of 16 images added to the training set. To measure performance, the median dice coefficient (DCE) was used to compare the model’s predicted segmentation to the analyst corrected annotation. The analyst provided estimates for annotation times.*Results Model performance increased from 92% to 98% DCE from the first iteration to the second iteration. Analyst annotation time decreased by approximately 20% and 35%, respectively, for images with model segmentations similar to the reference truth. Annotation time doubled for images where the model yielded a poor segmentation.*Conclusions Preliminary results suggest that the feedback loop accelerates the generation of well-curated training data. With each iteration, even building a model from scratch with few cases, we see improvements in the model's segmentations and reduction in annotation times. However, more iterations, and optimized processing techniques or model architectures, are needed to maintain a consistent performance throughout all images.*Clinical Relevance/Application A continuous feedback loop approach, iterating a continuous stream of data batches, can be used to build annotated training sets at a faster rate for training AI algorithms. This in turn aids in building more robust models for computer aided diagnosis tasks, or even tasks outside of medical imaging.

Model performance increased from 92% to 98% DCE from the first iteration to the second iteration. Analyst annotation time decreased by approximately 20% and 35%, respectively, for images with model segmentations similar to the reference truth. Annotation time doubled for images where the model yielded a poor segmentation.

A continuous feedback loop approach, iterating a continuous stream of data batches, can be used to build annotated training sets at a faster rate for training AI algorithms. This in turn aids in building more robust models for computer aided diagnosis tasks, or even tasks outside of medical imaging.