Reinhard Hameeteman MSC, Presenter: Nothing to Disclose

Maria Alejandra Zuluaga Valencia MSC, Abstract Co-Author: Nothing to Disclose

Moti Freiman, Abstract Co-Author: Nothing to Disclose

Aad Van Der Lugt MD, PhD, Abstract Co-Author: Nothing to Disclose

Theo Van Walsum, Abstract Co-Author: Nothing to Disclose

Manual stenosis grading in 3D CTA datasets is time-consuming and exhibits large inter-observer variabilities. Automated methods may replace manual measurements. However, this requires that their performance has been thoroughly and quantitatively evaluated. We therefore developed a publicly available, on-line framework for the evaluation of automated carotid bifurcation lumen segmentation and stenosis grading algorithms.

A multi-site, multi-vendor set of 56 clinical CTAs of the carotid bifurcation was acquired. For each CTA dataset the lumen and stenosis grade of one bifurcation was manually annotated by three observers. The manual annotations were combined to obtain ground truth lumen segmentations and stenosis gradings. Five evaluation measures for the lumen segmentation and stenosis grading were defined. Fifteen CTA datasets including ground truths for segmentation and stenosis grading have been made available for algorithm tuning. The remaining 41 datasets are available without the reference standard. Segmentation results and stenosis gradings could be uploaded to the framework website (http://cls2009.bigr.nl), evaluation measures are automatically determined by the evaluation framework software.

Around 100 research groups working on vascular segmentation algorithms have been invited to run their software on the datasets. To date, eight groups submitted results on lumen segmentation, and three groups on stenosis grading. Best result for lumen segmentation yields a mean surface distance of 0.18 mm, whereas the average interobserver surface distance is 0.12 mm. For the stenosis grading, the best results are around 15% difference, whereas the interobserver difference is 5%.

We provided a framework for the quantitative evaluation of automated stenosis grading methods. The on-line evaluation framework was successfully applied for evaluating several carotid image processing algorithms, and demonstrated that sub-millimeter accuracy is feasible for state-of-the-art CTA carotid lumen segmentation methods. The framework remains publicly available for future evaluation of novel stenosis grading methods.

Standardized evaluation of automated stenosis grading and lumen segmentation methods.

Hameeteman, R, Zuluaga Valencia, M, Freiman, M, Van Der Lugt, A, Van Walsum, T, A Standardized Evaluation Framework for Automated Carotid Bifurcation Lumen Segmentation and Stenosis Grading Methods.  Radiological Society of North America 2010 Scientific Assembly and Annual Meeting, November 28 - December 3, 2010 ,Chicago IL. http://archive.rsna.org/2010/9011998.html