Korean J Radiol.  2010 Apr;11(2):149-155. 10.3348/kjr.2010.11.2.149.

Observer Variability and the Performance between Faculties and Residents: US Criteria for Benign and Malignant Thyroid Nodules

• Kim SH
• Park CS
• Jung SL
• Kang BJ
• Kim JY
• Choi JJ
• Kim YI
• Oh JK
• Oh JS
• Kim H
• Jeong SH
• Yim HW

• KMID: 1783190
• DOI: http://doi.org/10.3348/kjr.2010.11.2.149

Abstract

OBJECTIVE
To evaluate the interobserver variability and performance in the interpretation of ultrasonographic (US) findings of thyroid nodules.
MATERIALS AND METHODS
72 malignant nodules and 61 benign nodules were enrolled as part of this study. Five faculty radiologists and four residents independently performed a retrospective analysis of the US images. The observers received one training session after the first interpretation and then performed a secondary interpretation. Agreement was analyzed by Cohen's kappa statistic. Degree of performance was analyzed using receiver operating characteristic (ROC) curves.
RESULTS
Agreement between the faculties was fair-to-good for all criteria; however, between residents, agreement was poor-to-fair. The area under the ROC curves was 0.72, 0.62, and 0.60 for the faculties, senior residents, and junior residents, respectively. There was a significant difference in performance between the faculties and the residents (p < 0.05). There was a significant increase in the agreement for some criteria in the faculties and the senior residents after the training session, but no significant increase in the junior residents.
CONCLUSION
Independent reporting of thyroid US performed by residents is undesirable. A continuous and specialized resident training is essential to enhance the degree of agreement and performance.

Keyword

Thyroid Nodule; Ultrasonography; Observer variation; Faculty; Internship and Residency

MeSH Terms

Adult
Aged
Clinical Competence/*statistics & numerical data
Diagnosis, Differential
Faculty, Medical/*statistics & numerical data
Female
Humans
Internship and Residency/*statistics & numerical data
Male
Middle Aged
Observer Variation
ROC Curve
Radiology/education
Retrospective Studies
Sensitivity and Specificity
Thyroid Gland/ultrasonography
Thyroid Neoplasms/*ultrasonography

Figure

• 

  Fig. 1 Right papillary carcinoma in 47-year-old woman. Axial (A) and sagittal (B) images representing case that showed moderate interobserver agreement for presence of microcalcifications. This case was described as having microcalcifications by all faculty radiologists and three out of four residents. One resident described it as having macrocalcifications.

• 

  Fig. 2 Left papillary carcinoma in 42-year-old man. Axial (A) and sagittal (B) images representing case that showed poor interobserver agreement for echotexture. At primary interpretation, three of five faculties and two of four residents described this case as having heterogeneous echotexture. Remaining observers described carcinoma as being homogeneous. At secondary interpretation, all faculties agreed on heterogeneous echotexture. In residents, however, no changes were seen.

• 

  Fig. 3 Left follicular adenoma in 18-year-old woman. Axial (A) and sagittal (B) images representing case that showed fair interobserver agreement for category. Faculty radiologists classified this case as indeterminate nodule at both primary and secondary interpretation. At primary interpretation, senior residents classified case as indeterminate (n = 1) and probably benign (n = 1). At secondary interpretation, however, they all classified case as indeterminate. Junior residents classified case as indeterminate and suspicious malignant at both primary and secondary interpretation.

• 

  Fig. 4 Receiver operating characteristic curves for faculty, senior residents, and junior residents. Area under curves was 0.72 for faculty, 0.62 for senior residents and 0.60 for junior residents. There is significant difference between area under the receiver operating characteristic curves values of faculties and residents.

Reference

1. Frates MC, Benson CB, Charboneau JW, Cibas ES, Clark OH, Coleman BG, et al. Management of thyroid nodules detected at US: Society of Radiologists in Ultrasound consensus conference statement. Radiology. 2005; 237:794–800. PMID: 16304103.

2. Kim EK, Park CS, Chung WY, Oh KK, Kim DI, Lee JT, et al. New sonographic criteria for recommending fine-needle aspiration biopsy of nonpalpable solid nodules of the thyroid. AJR Am J Roentgenol. 2002; 178:687–691. PMID: 11856699.

3. Chan BK, Desser TS, McDougall IR, Weigel RJ, Jeffrey RB Jr. Common and uncommon sonographic features of papillary thyroid carcinoma. J Ultrasound Med. 2003; 22:1083–1090. PMID: 14606565.

4. Iannuccilli JD, Cronan JJ, Monchik JM. Risk for malignancy of thyroid nodules as assessed by sonographic criteria: the need for biopsy. J Ultrasound Med. 2004; 23:1455–1464. PMID: 15498910.

5. Shimura H, Haraguchi K, Hiejima Y, Fukunari N, Fujimoto Y, Katagiri M, et al. Distinct diagnostic criteria for ultrasonographic examination of papillary thyroid carcinoma: a multicenter study. Thyroid. 2005; 15:251–258. PMID: 15785244.

6. Moon WJ, Jung SL, Lee JH, Na DG, Baek JH, Lee YH, et al. Benign and malignant thyroid nodules: US differentiation-multicenter retrospective study. Radiology. 2008; 247:762–770. PMID: 18403624.

7. Nam-Goong IS, Kim HY, Gong G, Lee HK, Hong SJ, Kim WB, et al. Ultrasonography-guided fine-needle aspiration of thyroid incidentaloma: correlation with pathological findings. Clin Endocrinol (Oxf). 2004; 60:21–22. PMID: 14678283.

8. Frates MC, Benson CB, Doubilet PM, Kunreuther E, Contreras M, Cibas ES, et al. Prevalence and distribution of carcinoma in patients with solitary and multiple thyroid nodules on sonography. J Clin Endocrinol Metab. 2006; 91:3411–3417. PMID: 16835280.

9. Thyroid Study Group of the Korean Society of Neuroradiology and Head and Neck Radiology (KSNRHNR). Thyroid gland: imaging diagnosis and intervention. 2008. 1st ed. Seoul: Ilchokak.

10. Fleiss JL. Statistical methods for rates and proportions. 1981. 2nd ed. New York: John Wiley & Sons Inc..

11. Cohen J. A coefficient of agreement for nominal scales. Educat Psychol Meas. 1960; 20:37–46.

12. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977; 33:159–174. PMID: 843571.

13. Wysoki MG, Nassar CJ, Koenigsberg RA, Novelline RA, Faro SH, Faerber EN. Head trauma: CT scan interpretation by radiology residents versus staff radiologists. Radiology. 1998; 208:125–128. PMID: 9646802.

14. Erly WK, Berger WG, Krupinski E, Seeger JF, Guisto JA. Radiology resident evaluation of head CT scan orders in the emergency department. AJNR Am J Neuroradiol. 2002; 23:103–107. PMID: 11827881.

15. Filippi CG, Schneider B, Burbank HN, Alsofrom GF, Linnell G, Ratkovits B. Discrepancy rates of radiology resident interpretations of on-call neuroradiology MR imaging studies. Radiology. 2008; 249:972–979. PMID: 19011191.

16. Carney E, Kempf J, DeCarvalho V, Yudd A, Nosher J. Preliminary interpretations of after-hours CT and sonography by radiology residents versus final interpretations by body imaging radiologists at a level 1 trauma center. AJR Am J Roentgenol. 2003; 181:367–373. PMID: 12876012.

17. Yavas US, Calisir C, Ozkan IR. The interobserver agreement between residents and experienced radiologists for detecting pulmonary embolism and DVT with using CT pulmonary angiography and indirect CT venography. Korean J Radiol. 2008; 9:498–502. PMID: 19039265.

18. Wienke JR, Chong WK, Fielding JR, Zou KH, Mittelstaedt CA. Sonographic features of benign thyroid nodules: interobserver reliability and overlap with malignancy. J Ultrasound Med. 2003; 22:1027–1031. PMID: 14606557.

19. Gur D, Bandos AI, Cohen CS, Hakim CM, Hardesty LA, Ganott MA, et al. The "laboratory" effect: comparing radiologists' performance and variability during prospective clinical and laboratory mammography interpretations. Radiology. 2008; 249:47–53. PMID: 18682584.