Abstract
This paper raises five methodological questions concerning Receiver operating characteristic (ROC) analysis: (1) can the ROC “confidence criterion” be applied in a valid, reliable way?; (2) can ROC deal with ambiguous findings?; (3) can ROC deal effectively with false-negative findings?; (4) are ROC curves susceptible to valid statistical testing?; and (5) are ROC results useful in choosing among alternative imaging modalities? A review of the evidence leads to six conclusions. First, using ROC, all radiological findings must be unambiguously scored as true-positive, true-negative, false-positive, or false-negative, often forcing arbitrary, procrustean choices on readers and evaluators. Second, ROC requires radiologists to report findings by confidence level on a consistent, reliable basis throughout a ROC experiment; something that seems unrealistic, given what is known about human performance in almost all perceptual tasks of comparable complexity. Third, as gathered during the typical experiment, ROC data are probably nominal, but treated as if ordinal (or even interval) data, leading to distorted results. Fourth, ROC does not deal effectively with false-negatives, despite their importance. Fifth, there is no satisfactory method for statistically testing the significance of observed differences between two ROC curves if they are based on nominal data. Finally, the artificial tasks required of radiologists in a ROC evaluation limit the usefulness of ROC results in choosing among the imaging modalities.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Chakraborty DP, Breatnach ES, Yester MV: Digital and conventional chest imaging: A modified ROC study of observer performance using simulated nodules. Radiology, 158:35–39, 1986
Getty DJ, Pickett RM, D’Orsi CJ, et al: Enhanced interpretation of diagnostic images. Invest Radiol 23:240–253, 1988
Kelsey CA, Mettler FA, Jr: ROC analysis can reveal best diagnostic method. Diagn Imag 11:155–161, 1989
Marioka C, Brown K, Hayrapetlan A, et al: ROC Analysis of Chest Radiographs Using Computed Radiography and Conventional Analog Films. The UCLA PACS Modules and Related Project, Medical Imaging Division, Department of Radiological Sciences, UCLA, 34–37, February 1989
McMahon H, Vyborny CT, Metz CE, et al: Digital chest radiography: Effect on diagnostic accuracy of hard copy, conventional video, and reversed gray scale video formats. Raiology 168:669–673, 1989
Seeley G, Newell JD: Use of Psychophysical Principles in the Design of a Total Digital Radiology Department. Radiologic Clinics of North America, Philadelphia, PA, Saunders, 23:341–348, 1985
Metz CE: Basic principles of ROC analysis. Semin Nucl Med VIII:283–298, 1978
Swets JA, Tanner WP: A decision-making theory of visual detection. Psychol Rev 61:401, 1954
Swets JA: ROC analysis applied to the evaluation of medical imaging Techniques. Invest Radiol 14:109–121, 1979
Kundel HL, Revesz G: The evaluation of radiographic techniques by observer tests: Pitfalls, problems and procedures. Invest Radiol 9:166–173, 1974
Swenson RG, Hessel SJ, Herman PG: Omissions in radiology: Faulty research or stringent reporting criteria? Radiology, 123:563–567, 1977
Truddenham WJ: Visual search, image organization and reader error in studies of the psychophysiology of roentgen image perception. Radiology 78:694–704, 1978
Metz CE: ROC methodology in radiologic imaging. Invest Radiol 26:120–123, 1986
Metz CE: Some practical issues of experimental design and data analysis in radiological ROC studies. Invest Radiol 24:234–245, 1989
Metz C, Wang P, Kronman HB, et al: A new approach for testing the significance of differences between ROC curves measured from correlated data, in Deconink F (ed) Information Processing In Medical Imaging. Hague, The Netherlands, Nijhoff, 1984, 432–445
Kundel HL: Peripheral vision, structured noise and film reader error. Radiology, 114:269–273, 1975
Swets JA: Form of empirical ROCs in discrimination and diagnostic tasks: Implications for theory and measurement of performance. Psychol Bull 99:181–198, 1986
Swets JA: Indices of discrimination or diagnostic accuracy: Their ROCs and implied models. Psychol Bull 99:100–117, 1988
Swets JA: Measuring the Accuracy of Diagnostic Systems. Science 1285–1293, 1988
Hanley JA, McNeil BJ: The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29–36, 1982
Hanley JA, McNeil BJ: Statistical approaches to analysis of receiver operating characteristic (ROC) curves. Med Decis Making 4:137–149, 1984
Revesz G, Kundel HL, Bonitatibus M, et al: The effect of verification on the assessment of imaging techniques. Invest Radiol 18:194–198, 1983
Roberts FS: Measurement Theory: With Applications to Decision-Making, Utility, and the Social Sciences, Reading, MA, Addison-Wesley, 1979, pp 4–34
Hanley JA, McNeil BJ: A method of comparing the areas under receiver operating characteristic curves derived from the same cases. Radiology 148:839–843, 1983
Hanley JA: Receiver operating characteristic (ROC) methodology: The state of the art. Crit Rev Diagn Imaging 29:307–335, 1989
Metz CE, Kronman HB: Statistical significance tests for binormal ROC curves. Math Psychol 22:218–243, 1980
Simel DL, Feussner JR, DeLong ER, et al: Intermediate, indeterminate, and uninterpretable diagnostic test results, Med Decis Making 7:107–114, 1987
Harrington MB, Miller KD: MTR-85W236, Results of the image quality evaluation conducted in conjunction with the 1984–1985 teleradiology field trial, McLean, VA: The MITRE Corporation, 1985
Siegel S: Non-parametric Statistics for the Behavioral Sciences. New York, NY, McGraw-Hill, 1956
Chakraborty DP: Free-response methodology: Alternative analysis and a new observer-performance experiment. Med Phys 174:873–881, 1990
Chakraborty DP: Maximum likelihood analysis of free-response receiver operating characteristic (FROC) data. Med Phys 16:561–568, 1989
Author information
Authors and Affiliations
Additional information
Adapted and reprinted with permission from Society of Photo-optical Engineers.
Rights and permissions
About this article
Cite this article
Harrington, M.B. Some methodological questions concerning receiver operating characteristic (ROC) analysis as a method for assessing image quality in radiology. J Digit Imaging 3, 211–218 (1990). https://doi.org/10.1007/BF03168117
Issue Date:
DOI: https://doi.org/10.1007/BF03168117