David William Piraino MD, Presenter: Medical Advisory Board, Agfa-Gevaert Group

Daniel W Palmer PhD, Abstract Co-Author: Nothing to Disclose

Nancy A. Obuchowski PhD, Abstract Co-Author: Nothing to Disclose

Michael JB Wang MD, Abstract Co-Author: Nothing to Disclose

Jennifer Bullen MSc, Abstract Co-Author: Nothing to Disclose

The phenomenon described as “the wisdom of crowds” posits that a group of individuals working independently on a problem on which they have varying levels of knowledge, can perform better than most of the individuals alone. Their solutions must be aggregated to produce a single solution that can outperform those from experts. Because each individual has some knowledge and some bias, the combination of their solutions should reinforce the knowledge (because it is the same), but cancel out the biases (because they are different). In order for a crowd to exhibit 'wisdom', they must attain the characteristics of diversity, and independence. Diversity ensures that many different approaches are considered. Independence prevents specific biases from spreading across the collective. Radiological diagnosis of images, including consults, violate both of these characteristics. We designed an experiment to satisfy these conditions to evaluate whether this phenomenon could be applied to radiology.

Seventy four musculoskeletal images with surgical proof or proof by follow up imaging were included. One third were normal, 1/3 were abnormal but easy to diagnosis, and 1/3 were abnormal but considered difficult to diagnosis. Twelve musculoskeletal radiologists marked the location of the abnormality and provide a differential diagnosis. Each radiologist evaluated each image separately without knowledge of other responses. A consensus diagnosis was calculated by a computer algorithm. Sensitivities and specificities were calculated for each reader and the algorithmic consensus. The sensitivity of the consensus was greater than the sensitivity of all 12 readers and the consensus specificity was great than 10 of the readers and equal to the other 2.

The algorithmic consensus derived from independent readers had a greater sensitivity than all readers (statistically significant for 4 readers using McNemar’s test at 0.05 level) and had a specificity greater than or equal to all readers (statistically significantly for 4 readers using McNemar’s test).

This experiment demonstrates that algorithmic aggregation of individual expert image diagnoses can perform better than individual experts.

Piraino, D, Palmer, D, Obuchowski, N, Wang, M, Bullen, J, Leveraging "The Wisdom of Crowds" in Collaborative Image Diagnosis.  Radiological Society of North America 2013 Scientific Assembly and Annual Meeting, December 1 - December 6, 2013 ,Chicago IL. http://archive.rsna.org/2013/13013893.html