Korean J Radiol.  2020 Mar;21(3):369-376. 10.3348/kjr.2019.0581.

Computer-Aided Diagnosis System for the Evaluation of Thyroid Nodules on Ultrasonography: Prospective Non-Inferiority Study according to the Experience Level of Radiologists

• Chung SR
• Baek JH
• Lee MK
• Ahn Y
• Choi YJ
• Sung TY
• Song DE
• Kim TY
• Lee JH

• KMID: 2470761
• DOI: http://doi.org/10.3348/kjr.2019.0581

Abstract

OBJECTIVE
To determine whether a computer-aided diagnosis (CAD) system for the evaluation of thyroid nodules is non-inferior to radiologists with different levels of experience.
MATERIALS AND METHODS
Patients with thyroid nodules with a decisive diagnosis of benign or malignant nodule were consecutively enrolled from November 2017 to September 2018. Three radiologists with different levels of experience (1 month, 4 years, and 7 years) in thyroid ultrasound (US) reviewed the thyroid US with and without using the CAD system. Statistical analyses included non-inferiority testing of the diagnostic accuracy for malignant thyroid nodules between the CAD system and the three radiologists with a non-inferiority margin of 10%, comparison of the diagnostic performance, and the added value of the CAD system to the radiologists.
RESULTS
Altogether, 197 patients were included in the study cohort. The diagnostic accuracy of the CAD system (88.48%, 95% confidence interval [CI] = 82.65-92.53) was non-inferior to that of the radiologists with less experience (1 month and 4 year) of thyroid US (83.03%, 95% CI = 76.52-88.02; p < 0.001), whereas it was inferior to that of the experienced radiologist (7 years) (95.76%, 95% CI = 91.37-97.96; p = 0.138). The sensitivity and negative predictive value of the CAD system were significantly higher than those of the less-experienced radiologists were, whereas no significant difference was found with those of the experienced radiologist. A combination of US and the CAD system significantly improved sensitivity and negative predictive value, although the specificity and positive predictive value deteriorated for the less-experienced radiologists.
CONCLUSION
The CAD system may offer support for decision-making in the diagnosis of malignant thyroid nodules for operators who have less experience with thyroid US.

Keyword

Computer-aided diagnosis; Thyroid nodule; Thyroid cancer; Ultrasonography

MeSH Terms

Cohort Studies
Diagnosis*
Humans
Prospective Studies*
Sensitivity and Specificity
Thyroid Gland*
Thyroid Neoplasms
Thyroid Nodule*
Ultrasonography*

Figure

• 

  Fig. 1 Representative case of malignant thyroid nodule. US image (A) and automatically calculated mass contour, US features, and diagnosis presented by CAD system (B). Both CAD system and radiologist diagnosed it as malignant nodule. CAD system performed excellent segmentation of thyroid nodule. CAD system and radiologist demonstrated concordance regarding US characteristics of composition (solid), shape (ovoid-to-round), orientation (non-parallel), echogenicity (hypoechogenicity), and margin (spiculated). CAD = computer-aided diagnosis, US = ultrasound

• 

  Fig. 2 Representative case of benign thyroid nodule. US image (A) and automatically calculated mass contour, US features, and diagnosis presented by CAD system (B). Both CAD system and radiologist diagnosed it as benign nodule. CAD system performed satisfactory segmentation of thyroid nodule. CAD system and radiologist demonstrated concordance regarding US characteristics of composition (partially cystic), shape (ovoid-to-round), orientation (parallel), echogenicity (hyperechoic/isoechoic), and margin (well-defined).

Reference

1. Guth S, Theune U, Aberle J, Galach A, Bamberger CM. Very high prevalence of thyroid nodules detected by high frequency (13 MHz) ultrasound examination. Eur J Clin Invest. 2009; 39:699–706.

2. Tan GH, Gharib H. Thyroid incidentalomas: management approaches to nonpalpable nodules discovered incidentally on thyroid imaging. Ann Intern Med. 1997; 126:226–231.

3. Haugen BR, Alexander EK, Bible KC, Doherty GM, Mandel SJ, Nikiforov YE, et al. 2015 American Thyroid Association Management Guidelines for adult patients with thyroid nodules and differentiated thyroid cancer: the American Thyroid Association Guidelines task force on thyroid nodules and differentiated thyroid cancer. Thyroid. 2016; 26:1–133.

4. Frates MC, Benson CB, Doubilet PM, Kunreuther E, Contreras M, Cibas ES, et al. Prevalence and distribution of carcinoma in patients with solitary and multiple thyroid nodules on sonography. J Clin Endocrinol Metab. 2006; 91:3411–3417.

5. Nam-Goong IS, Kim HY, Gong G, Lee HK, Hong SJ, Kim WB, et al. Ultrasonography-guided fine-needle aspiration of thyroid incidentaloma: correlation with pathological findings. Clin Endocrinol (Oxf). 2004; 60:21–28.

6. Papini E, Guglielmi R, Bianchini A, Crescenzi A, Taccogna S, Nardi F, et al. Risk of malignancy in nonpalpable thyroid nodules: predictive value of ultrasound and color-Doppler features. J Clin Endocrinol Metab. 2002; 87:1941–1946.

7. Choi SH, Kim EK, Kwak JY, Kim MJ, Son EJ. Interobserver and intraobserver variations in ultrasound assessment of thyroid nodules. Thyroid. 2010; 20:167–172.

8. Moon WJ, Jung SL, Lee JH, Na DG, Baek JH, Lee YH, et al. Benign and malignant thyroid nodules: US differentiation—Multicenter retrospective study. Radiology. 2008; 247:762–770.

9. Park CS, Kim SH, Jung SL, Kang BJ, Kim JY, Choi JJ, et al. Observer variability in the sonographic evaluation of thyroid nodules. J Clin Ultrasound. 2010; 38:287–293.

10. Kim SH, Park CS, Jung SL, Kang BJ, Kim JY, Choi JJ, et al. Observer variability and the performance between faculties and residents: US criteria for benign and malignant thyroid nodules. Korean J Radiol. 2010; 11:149–155.

11. Acharya UR, Sree SV, Krishnan MM, Molinari F, Zieleźnik W, Bardales RH, et al. Computer-aided diagnostic system for detection of Hashimoto thyroiditis on ultrasound images from a Polish population. J Ultrasound Med. 2014; 33:245–253.

12. Chang Y, Paul AK, Kim N, Baek JH, Choi YJ, Ha EJ, et al. Computer-aided diagnosis for classifying benign versus malignant thyroid nodules based on ultrasound images: a comparison with radiologist-based assessments. Med Phys. 2016; 43:554.

13. Choi YJ, Baek JH, Park HS, Shim WH, Kim TY, Shong YK, et al. A computer-aided diagnosis system using artificial intelligence for the diagnosis and characterization of thyroid nodules on ultrasound: initial clinical assessment. Thyroid. 2017; 27:546–552.

14. Li LN, Ouyang JH, Chen HL, Liu DY. A computer aided diagnosis system for thyroid disease using extreme learning machine. J Med Syst. 2012; 36:3327–3337.

15. Lim KJ, Choi CS, Yoon DY, Chang SK, Kim KK, Han H, et al. Computer-aided diagnosis for the differentiation of malignant from benign thyroid nodules on ultrasonography. Acad Radiol. 2008; 15:853–858.

16. Yoo YJ, Ha EJ, Cho YJ, Kim HL, Han M, Kang SY. Computer-aided diagnosis of thyroid nodules via ultrasonography: initial clinical experience. Korean J Radiol. 2018; 19:665–672.

17. Yu Q, Jiang T, Zhou A, Zhang L, Zhang C, Xu P. Computer-aided diagnosis of malignant or benign thyroid nodes based on ultrasound images. Eur Arch Otorhinolaryngol. 2017; 274:2891–2897.

18. Gao L, Liu R, Jiang Y, Song W, Wang Y, Liu J, et al. Computer-aided system for diagnosing thyroid nodules on ultrasound: a comparison with radiologist-based clinical assessments. Head Neck. 2018; 40:778–783.

19. Jeong EY, Kim HL, Ha EJ, Park SY, Cho YJ, Han M. Computer-aided diagnosis system for thyroid nodules on ultrasonography: diagnostic performance and reproducibility based on the experience level of operators. Eur Radiol. 2019; 29:1978–1985.

20. Liu JP, Hsueh HM, Hsieh E, Chen JJ. Tests for equivalence or non-inferiority for paired binary data. Stat Med. 2002; 21:231–245.

21. Kim HL, Ha EJ, Han M. Real-world performance of computer-aided diagnosis system for thyroid nodules using ultrasonography. Ultrasound Med Biol. 2019; 45:2672–2678.

22. Zhao WJ, Fu LR, Huang ZM, Zhu JQ, Ma BY. Effectiveness evaluation of computer-aided diagnosis system for the diagnosis of thyroid nodules on ultrasound: a systematic review and meta-analysis. Medicine (Baltimore). 2019; 98:e16379.

23. Gitto S, Grassi G, De Angelis C, Monaco CG, Sdao S, Sardanelli F, et al. A computer-aided diagnosis system for the assessment and characterization of low-to-high suspicion thyroid nodules on ultrasound. Radiol Med. 2019; 124:118–125.