OBSERVER PERFORMANCE METHODS FOR DIAGNOSTIC IMAGING
Foundations, Modeling, and Applications with R-Based Examples
A book with the above title is has been published by Taylor-Francis. A summary of the book chapters is attached.
ABSTRACT: There exist methods of objectively evaluating the performance of radiologists based on subjective assessments of likelihood of disease on a set of cases. They are termed Receiver Operating Characteristic (ROC) analyses. There is need to evaluate new imaging advances and extensive software tools have been developed, but almost all of them only analyze ROC data where the radiologist rates each image for confidence in presence of disease "somewhere in the image". Unlike ROC, free-response ROC (FROC) methodology accounts for disease location. For over three decades the author has been at the forefront of FROC research. Since 2004 he has distributed free, open-source, cross-platform R-software, which, including an earlier Windows version, have been used on over 107 research publications. This book is aimed at users of the author's software, and others, who seek a better understanding of the methods used to assess imaging systems, radiologists or computer-aided detection (CAD) performance. Most existing books on the subject are intended for statisticians and deal minimally with CAD and the FROC paradigm. Assuming little statistical background, the book covers ROC-FROC analysis from a basic level to recent advances. Fundamental concepts are explained using R-coded examples. The book is unique in that it has a substantial online component (http://www.expertcadanalytics.com), which is continually updated to reflect current research advances. Following an introductory chapter, which lays the groundwork, the book consists of four parts. Part (A) covers the basics of the ROC paradigm. Part (B) covers statistical methods used to compare performance using the ROC paradigm, including sample size estimation. Part (C) covers the FROC paradigm. Part (D) covers advanced topics including comparing the performance of a designer-level CAD algorithm to a group of radiologists interpreting the same cases. Another unique feature is the emphasis on data modeling. While non-parametric methods have their place, they yield minimal insight into factors limiting performance. Key-chapters (#16 - #18) demonstrate a radiological search model-based fitting of 236 ROC datasets, yielding important insights into what is limiting radiologist performance. Besides educating the user community, the methods detailed in this book will spur improved methods of designing CAD algorithms and training, evaluating and certifying radiologists.
The entire online appendix to my book, which has lots of examples, is on GitHub.