J Korean Med Sci.  2023 Sep;38(38):e319. 10.3346/jkms.2023.38.e319.

GPTZero Performance in Identifying Artificial Intelligence-Generated Medical Texts: A Preliminary Study

• Habibzadeh F

• KMID: 2546223
• DOI: http://doi.org/10.3346/jkms.2023.38.e319

Abstract

Background
With emergence of chatbots to help authors with scientific writings, editors should have tools to identify artificial intelligence-generated texts. GPTZero is among the first websites that has sought media attention claiming to differentiate machine-generated from human-written texts.
Methods
Using 20 text pieces generated by ChatGPT in response to arbitrary questions on various topics in medicine and 30 pieces chosen from previously published medical articles, the performance of GPTZero was assessed.
Results
GPTZero had a sensitivity of 0.65 (95% confidence interval, 0.41–0.85); specificity, 0.90 (0.73–0.98); accuracy, 0.80 (0.66–0.90); and positive and negative likelihood ratios, 6.5 (2.1–19.9) and 0.4 (0.2–0.7), respectively.
Conclusion
GPTZero has a low false-positive (classifying a human-written text as machinegenerated) and a high false-negative rate (classifying a machine-generated text as human-written).

Keyword

Scientific Writing; Journalism; Artificial Intelligence; Classification; Sensitivity and Specificity

Reference

1. Habibzadeh F. The future of scientific journals: the rise of UniAI. Learn Publ. 2023; 36(2):326–330.

2. Doskaliuk B, Zimba O. Beyond the keyboard: academic writing in the era of ChatGPT. J Korean Med Sci. 2023; 38(26):e207. PMID: 37401498.

3. Benichou L, ChatGPT . Rôle de l'utilisation de l'intelligence artificielle ChatGPT dans la rédaction des articles scientifiques médicaux The Role of Using ChatGPT AI in Writing Medical Scientific Articles. J Stomatol Oral Maxillofac Surg. 2023; 101456. PMID: 36966950.

4. Curtis N, ChatGPT . To ChatGPT or not to ChatGPT? The impact of artificial intelligence on academic publishing. Pediatr Infect Dis J. 2023; 42(4):275. PMID: 36757192.

5. King MR, ChatGPT . A Conversation on artificial intelligence, chatbots, and plagiarism in higher education. Cell Mol Bioeng. 2023; 16(1):1–2. PMID: 36660590.

6. ChatGPT Generative Pre-trained Transformer. Zhavoronkov A. Rapamycin in the context of Pascal’s Wager: generative pre-trained transformer perspective. Oncoscience. 2022; 9:82–84. PMID: 36589923.

7. International Committee of Medical Journal Editors (ICMJE). Recommendations for the conduct, reporting, editing, and publication of scholarly work in medical journals. Updated 2023. Accessed June 12, 2023. https://www.icmje.org/icmje-recommendations.pdf .

8. Zielinski C, Winker MA, Aggarwal R, Ferris LE, Heinemann M, Lapeña JF, et al. Chatbots, generative AI, and scholarly manuscripts. WAME recommendations on chatbots and generative artificial intelligence in relation to scholarly publications. Updated 2023. Accessed June 2, 2023. https://wame.org/page3.php?id=106 .

9. Kirmani AR. Fantastic (AI) bots and how to catch them. MRS Bull. 2023; 48(4):314–318.

10. Habibzadeh F. The acceptable text similarity level in manuscripts submitted to scientific journals. J Korean Med Sci. 2023; 38(31):e240. PMID: 37550808.

11. Bekaryssova D, Joshi M, Gupta L, Yessirkepov M, Gupta P, Zimba O, et al. Knowledge and perceptions of reactive arthritis diagnosis and management among healthcare workers during the COVID-19 pandemic: online survey. J Korean Med Sci. 2022; 37(50):e355. PMID: 36573387.

12. Casagrande A, Fabris F, Girometti R. Fifty years of Shannon information theory in assessing the accuracy and agreement of diagnostic tests. Med Biol Eng Comput. 2022; 60(4):941–955. PMID: 35195818.

13. Choi JW, Kim HJ, Kim JB, Lee S, Lim C, Chang BC, et al. Early and two-year outcomes after sutureless and conventional aortic valve replacement: a nationwide population-based study. J Korean Med Sci. 2021; 36(9):e57. PMID: 33686809.

14. Chumakov K, Benn CS, Aaby P, Kottilil S, Gallo R. Can existing live vaccines prevent COVID-19? Science. 2020; 368(6496):1187–1188. PMID: 32527819.

15. Gasparyan AY, Ayvazyan L, Mukanova U, Yessirkepov M, Kitas GD. Scientific hypotheses: writing, promoting, and predicting implications. J Korean Med Sci. 2019; 34(45):e300. PMID: 31760713.

16. Gasparyan AY, Kumar AB, Yessirkepov M, Zimba O, Nurmashev B, Kitas GD. Global health strategies in the face of the COVID-19 pandemic and other unprecedented threats. J Korean Med Sci. 2022; 37(22):e174. PMID: 35668684.

17. Gupta L, Gasparyan AY, Misra DP, Agarwal V, Zimba O, Yessirkepov M. Information and misinformation on COVID-19: a cross-sectional survey study. J Korean Med Sci. 2020; 35(27):e256. PMID: 32657090.

18. Habibzadeh F. A bird’s eye view of science publishing and editing in Iran. Eur Sci Ed. 2006; 32(4):98–100.

19. Habibzadeh F. Statistical data editing in scientific articles. J Korean Med Sci. 2017; 32(7):1072–1076. PMID: 28581261.

20. Habibzadeh F. Open access journals in the Middle East and Iran. J Korean Med Sci. 2019; 34(16):e123. PMID: 31020814.

21. Habibzadeh F, Winker M. Duplicate publication and plagiarism: causes and cures. Notf Rettmed. 2009; 12(6):415–418.

22. Habibzadeh F, Chumakov K, Sajadi MM, Yadollahie M, Stafford K, Simi A, et al. Use of oral polio vaccine and the incidence of COVID-19 in the world. PLoS One. 2022; 17(3):e0265562. PMID: 35298546.

23. Habibzadeh F, Habibzadeh P, Shakibafard A, Saidi F. Predicting the outcome of asymptomatic univesicular liver hydatids: diagnostic accuracy of unenhanced CT. Eur Radiol. 2021; 31(8):5812–5817. PMID: 33452910.

24. Habibzadeh F, Habibzadeh P, Yadollahie M. The apparent prevalence, the true prevalence. Biochem Med (Zagreb). 2022; 32(2):020101. PMID: 35799992.

25. Habibzadeh F, Habibzadeh P, Yadollahie M, Roozbehi H. On the information hidden in a classifier distribution. Sci Rep. 2021; 11(1):917. PMID: 33441644.

26. Habibzadeh F, Sajadi MM, Chumakov K, Yadollahie M, Kottilil S, Simi A, et al. COVID-19 infection among women in Iran exposed vs unexposed to children who received attenuated poliovirus used in oral polio vaccine. JAMA Netw Open. 2021; 4(11):e2135044. PMID: 34817583.

27. Habibzadeh F, Yadollahie M, Simi A. Use of oral polio vaccine and the global incidence of mother-to-child human immunodeficiency virus transmission. Front Public Health. 2022; 10:878298. PMID: 35812500.

28. Habibzadeh P, Dastsooz H, Eshraghi M, Łos MJ, Klionsky DJ, Ghavami S. Autophagy: the potential link between SARS-CoV-2 and cancer. Cancers (Basel). 2021; 13(22):5721. PMID: 34830876.

29. Habibzadeh P, Mofatteh M, Silawi M, Ghavami S, Faghihi MA. Molecular diagnostic assays for COVID-19: an overview. Crit Rev Clin Lab Sci. 2021; 58(6):385–398. PMID: 33595397.

30. Kim YK, Minn D, Song DY, Lee CH, Ryoo NH, Jeon CH, et al. Prevalence of SARS-CoV-2 antibody in 2,935 healthcare workers at 6 major hospitals, Daegu, Korea. J Korean Med Sci. 2021; 36(43):e294. PMID: 34751011.

31. Krakowska A, Cedzyński M, Wosiak A, Swiechowski R, Krygier A, Tkaczyk M, et al. Toll-like receptor (TLR2, TLR4) polymorphisms and their influence on the incidence of urinary tract infections in children with and without urinary tract malformation. Cent Eur J Immunol. 2022; 47(3):260–266. PMID: 36817265.

32. Malek-Hosseini SA, Habibzadeh F, Nikeghbalian S. Shiraz organ transplant center: the largest liver transplant center in the world. Transplantation. 2019; 103(8):1523–1525. PMID: 31348432.

33. Saidi F, Habibzadeh F. The non-operative management of asymptomatic liver hydatids: ending echinococcophobia. J Gastrointest Surg. 2018; 22(3):486–495. PMID: 29119531.

34. Singh AV, Yadav VS, Chauhan DS, Singh SV. Mycobacterium bovis induced human tuberculosis in India: current status, challenges & opportunities. Indian J Med Res. 2022; 156(1):21–30. PMID: 36510895.

35. Vessal K, Habibzadeh F. Rules of the game of scientific writing: fair play and plagiarism. Lancet. 2007; 369(9562):641.

36. Habibzadeh F, Habibzadeh P. The likelihood ratio and its graphical representation. Biochem Med (Zagreb). 2019; 29(2):020101. PMID: 31015780.

37. Habibzadeh F, Habibzadeh P, Yadollahie M. On determining the most appropriate test cut-off value: the case of tests with continuous results. Biochem Med (Zagreb). 2016; 26(3):297–307. PMID: 27812299.

38. MedCalc Software Ltd. Diagnostic test evaluation calculator: version 22.007. Updated 2023. Accessed June 24, 2023. https://www.medcalc.org/calc/diagnostic_test.php .

39. Ahmad N, Murugesan S, Kshetri N. Generative artificial intelligence and the education sector. Computer. 2023; 56(6):72–76.