aka the JAFROC website

How to conduct a free-response study

Free response data collection: Free response data consists of mark-rating pairs (responses) acquired under conditions of free-search. Each mark represents a region that was suspicious for disease (e.g., cancer). The radiologist must not know how many true lesions are present in the image, where they could be present, and the number of marks (0 or more) must not be limited by the investigator. For each mark the radiologist must give a confidence level (degree of suspicion) that the region represents a lesion. The rating is any number, with higher numbers representing greater confidence. See ReadMe file for more details.

Collecting the mark rating data: for films use an acrylic overlay properly aligned to the image and have the radiologist mark the location(s) and write the rating next to each mark. For digital images keep the interface simple and avoid having distracting cues on the image. Location information could be recorded at pixel level resolution with digital images. Consult with your radiologist to determine the most appropriate method. Some radiologists prefer to indicate one of several pre-specified clinically related regions (regions of interest). Save the location information and the locations of actual lesions - they may be need for future analysis.

Scoring the mark rating data involves classifying each mark as either a "lesion localization" LL (colloquially a true positive, TP), or a "non-lesion localization" NL (a false positive, FP), according to its proximity to a real lesion. Talk to the radiologist to establish what constitutes a clinically acceptable proximity criterion.

Multiple views, multiple CT slices, or multiple modalities per case: in addition to the location information, record the view on which the mark occurred, the CT slice number, the modality, etc.

Creator: Dev P. Chakraborty, PhD