Samuel George Armato PhD, Presenter: Shareholder, Hologic, Inc (R2 Technology, Inc)

Heber M. MacMahon MD, Abstract Co-Author: Consultant, Riverain Medical Research support, MEDIAN Technologies Stockholder, Hologic, Inc (R2 Technology, Inc)

Denise Ru Aberle MD, Abstract Co-Author: Nothing to Disclose

Ella A. Kazerooni MD, Abstract Co-Author: Consultant, General Electric Company Consultant, Vital Images, Inc

Edwin J.R. Van Beek MD, PhD, Abstract Co-Author: Consultant, Schering-Plough Corporation Medical Advisory Board, EDDA Technology, Inc Medical Advisory Board, Vital Images, Inc Speakers Bureau, Koninklijke Philips Electronics NV Research support, General Electric Company Research support, Siemens AG Medical Advisory Board, VIDA Diagnostics

David F. Yankelevitz MD, Abstract Co-Author: Consultant, PneumRx, Inc Shareholder, PneumRx, Inc

Rachael Y. Roberts MD, Abstract Co-Author: Nothing to Disclose

Geoffrey McLennan MD, PhD, Abstract Co-Author: Nothing to Disclose

Michael F. McNitt-Gray PhD, Abstract Co-Author: Nothing to Disclose

Charles R. Meyer PhD, Abstract Co-Author: Nothing to Disclose

Anthony P. Reeves PhD, Abstract Co-Author: Nothing to Disclose

Barbara Yoder Croft PhD, Abstract Co-Author: Nothing to Disclose

Laurence P. Clarke PhD, Abstract Co-Author: Nothing to Disclose

et al, Abstract Co-Author: Nothing to Disclose

et al, Abstract Co-Author: Nothing to Disclose

Studies that evaluate the lung-nodule-detection performance of radiologists or computerized methods depend on an initial inventory of the nodules within the thoracic images (the "truth"). The purpose of this study was to analyze (1) variability in the "truth" defined by different combinations of experienced thoracic radiologists and (2) variability in the performance of other experienced thoracic radiologists based on these definitions of "truth" in the context of lung nodule detection on computed tomography (CT) scans.

Twenty-five thoracic CT scans were reviewed independently by four thoracic radiologists. During this process, the radiologists independently marked lesions they considered to be nodules greater than or equal to 3 mm in diameter. After all scans were evaluated, sets of "true" nodules (a "truth" assessment) were derived retrospectively from the nodules marked by different combinations of two of the four radiologists, and the nodule-detection performance of the other two radiologists was evaluated based on this two-radiologist "truth" assessment. "Truth" assessments were obtained from all six combinations of two radiologists and comprised two variants: one based on the logical AND combination of the two radiologists' nodule marks and the other based on a logical OR combination of the two radiologists' nodule marks.

The number of "true" nodules across the 12 different "truth" assessments ranged from 17-77 (mean: 48.8±21.9). The nodule-detection sensitivity of the radiologists ranged from 24.7-100.0% (mean: 68.0%±23.0%) with a false-positive rate that ranged from 0.08-1.72 per case (mean: 0.74±0.56 per case).

Substantial variability exists across radiologists in the task of lung nodule identification in CT scans. The definition of "truth" on which lung nodule detection studies are based must be robust, since even experienced thoracic radiologists may not "perform" well when measured against the "truth" established by a two-radiologist panel.

The method of establishing "truth" for studies that evaluate the identification of lung nodules in CT scans must be given proper consideration and clearly understood.

Armato, S, MacMahon, H, Aberle, D, Kazerooni, E, Van Beek, E, Yankelevitz, D, Roberts, R, McLennan, G, McNitt-Gray, M, Meyer, C, Reeves, A, Croft, B, Clarke, L, et al, , et al, , "Truth" and Radiologist Performance in the Detection of Lung Nodules.  Radiological Society of North America 2007 Scientific Assembly and Annual Meeting, November 25 - November 30, 2007 ,Chicago IL. http://archive.rsna.org/2007/5014291.html