Berkman Sahiner PhD, Presenter: Nothing to Disclose

Aria Pezeshk PhD, Abstract Co-Author: Nothing to Disclose

Xin He PhD, Abstract Co-Author: Nothing to Disclose

Weijie Chen PhD, Abstract Co-Author: Nothing to Disclose

Rongping Zeng PhD, Abstract Co-Author: Nothing to Disclose

Nicholas Petrick PhD, Abstract Co-Author: Nothing to Disclose

Frank W. Samuelson PhD, Abstract Co-Author: Nothing to Disclose

To investigate the feasibility of modeling to predict radiologists’ computer-aided diagnosis (CADx)-assisted ratings based on their unassisted ratings and standalone CADx scores.

Our data set consisted of radiologists’ malignancy ratings on a 101-point scale for breast masses on mammograms and ultrasound images without and with the use of a multi-modality CADx system. Ten breast radiologists provided ratings for 67 breast masses (35 malignant and 32 benign) in a fully-crossed design. We used radiologists’ unassisted ratings and standalone computer scores as predictor variables in linear regression models to predict the assisted ratings. We compared two methods for modeling: Method one used a single radiologist’s data to predict another radiologist’s ratings, and method two used the average ratings of a training set of radiologists to predict the average ratings of the remaining set of test radiologists. In method two, the data set was randomly partitioned into sets of five training and five test radiologists 200 times. We used a ten-fold cross validation technique to partition the cases into training and test sets for each method. Separate models were developed for malignant and benign masses. The performance of each model was measured using the correlation coefficient (CC) between the predicted and true assisted ratings.

For models trained with a single radiologist, the average CC values were 0.86 (range:0.57-0.98) for malignant and 0.88 (range: 0.59-0.98) for benign masses. In comparison, for models trained with average ratings, the average CC values were 0.95 (range: 0.89-0.98) for malignant and 0.95 (range: 0.92-0.97) for benign masses. The average area under the receiver operating curve (AUC) obtained using the predicted and true average CAD-assisted ratings were 0.987 (se: 0.008), and 0.979 (se: 0.008), respectively. In comparison, the average AUC value obtained from the averaged unassisted ratings was 0.959 (se: 0.014).

Averaging the ratings of a group of radiologists allowed for the construction of accurate models for the prediction of ratings of a different group of readers. Using single-radiologist data for prediction resulted in lower accuracy.

Most CADx systems are optimized based on standalone performance. Modeling radiologist-CAD interaction may result in improved optimization based on the predicted CAD-assisted radiologist performance.

Sahiner, B, Pezeshk, A, He, X, Chen, W, Zeng, R, Petrick, N, Samuelson, F, Modeling for Prediction of CAD-assisted Ratings of Radiologists in Breast Mass Characterization: A Feasibility Study.  Radiological Society of North America 2014 Scientific Assembly and Annual Meeting, - ,Chicago IL. http://archive.rsna.org/2014/14045557.html