Healthc Inform Res.  2019 Oct;25(4):305-312. 10.4258/hir.2019.25.4.305.

Machine Learning-Based Prediction of Korean Triage and Acuity Scale Level in Emergency Department Patients

• Choi SW
• Ko T
• Hong KJ
• Kim KH

• KMID: 2462229
• DOI: http://doi.org/10.4258/hir.2019.25.4.305

Abstract

OBJECTIVES
Triage is a process to accurately assess and classify symptoms to identify and provide rapid treatment to patients. The Korean Triage and Acuity Scale (KTAS) is used as a triage instrument in all emergency centers. The aim of this study was to train and compare machine learning models to predict KTAS levels.
METHODS
This was a cross-sectional study using data from a single emergency department of a tertiary university hospital. Information collected during triage was used in the analysis. Logistic regression, random forest, and XGBoost were used to predict the KTAS level.
RESULTS
The models with the highest area under the receiver operating characteristic curve (AUROC) were the random forest and XGBoost models trained on the entire dataset (AUROC = 0.922, 95% confidence interval 0.917-0.925 and AUROC = 0.922, 95% confidence interval 0.918-0.925, respectively). The AUROC of the models trained on the clinical data was higher than that of models trained on text data only, but the models trained on all variables had the highest AUROC among similar machine learning models.
CONCLUSIONS
Machine learning can robustly predict the KTAS level at triage, which may have many possibilities for use, and the addition of text data improves the predictive performance compared to that achieved by using structured data alone.

Keyword

Triage; Hospital Emergency Service; Machine Learning; Natural Language Processing

MeSH Terms

Cross-Sectional Studies
Dataset
Emergencies*
Emergency Service, Hospital*
Forests
Humans
Logistic Models
Machine Learning
Natural Language Processing
ROC Curve
Triage*

Figure

• 

  Figure 1 Receiver operating characteristic curve for selected machine learning models. AUROC: area under the receiver operating characteristic curve, LR: logistic regression, RF: random forest, XGB: XGBoost.

Reference

1. National Emergency Medical Center. Emergency Medical Statistics Annual Report 2018 [Internet]. Seoul, Korea: National Emergency Medical Center;c2019. cited at 2019 Oct 18. Available from: https://www.e-gen.or.kr/nemc/statistics_annual_report.do.

2. Park J, Lim T. Korean Triage and Acuity Scale (KTAS). J Korean Soc Emerg Med. 2017; 28(6):547–551.

3. Ryu JH, Min MK, Lee DS, Yeom SR, Lee SH, Wang IJ, et al. Changes in relative importance of the 5-level triage system, Korean Triage and Acuity Scale, for the disposition of emergency patients induced by forced reduction in its level number: a multi-center registry-based retrospective cohort study. J Korean Med Sci. 2019; 34(14):e114.

4. Kwon H, Kim YJ, Jo YH, Lee JH, Lee JH, Kim J, et al. The Korean Triage and Acuity Scale: associations with admission, disposition, mortality and length of stay in the emergency department. Int J Qual Health Care. 2019; 31(6):449–455.

5. Hinson JS, Martinez DA, Cabral S, George K, Whalen M, Hansoti B, et al. Triage performance in emergency medicine: a systematic review. Ann Emerg Med. 2019; 74(1):140–152.

6. KTAS education guide [Internet]. Seoul, Korea: Korean Triage and Acuity Scale;c2019. cited at 2019 Oct 17. Available from: http://www.ktas.org/education/info.php.

7. Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An interpretable machine learning model for accurate prediction of sepsis in the ICU. Crit Care Med. 2018; 46(4):547–553.

8. Gupta A, Liu T, Shepherd S, Paiva W. Using statistical and machine learning methods to evaluate the prognostic accuracy of SIRS and qSOFA. Healthc Inform Res. 2018; 24(2):139–147.

9. Park JH, Shin SD, Song KJ, Hong KJ, Ro YS, Choi JW, et al. Prediction of good neurological recovery after out-of-hospital cardiac arrest: a machine learning analysis. Resuscitation. 2019; 142:127–135.

10. Stewart J, Sprivulis P, Dwivedi G. Artificial intelligence and machine learning in emergency medicine. Emerg Med Australas. 2018; 30(6):870–874.

11. Hong WS, Haimovich AD, Taylor RA. Predicting hospital admission at emergency department triage using machine learning. PLoS One. 2018; 13(7):e0201016.

12. Goto T, Camargo CA Jr, Faridi MK, Freishtat RJ, Hasegawa K. Machine learning-based prediction of clinical outcomes for children during emergency department triage. JAMA Netw Open. 2019; 2(1):e186937.

13. Kwon JM, Lee Y, Lee Y, Lee S, Park H, Park J. Validation of deep-learning-based triage and acuity score using a large national dataset. PLoS One. 2018; 13(10):e0205836.

14. Raita Y, Goto T, Faridi MK, Brown DFM, Camargo CA Jr, Hasegawa K. Emergency department triage prediction of clinical outcomes using machine learning models. Crit Care. 2019; 23(1):64.

15. Levin S, Toerper M, Hamrock E, Hinson JS, Barnes S, Gardner H, et al. Machine-learning-based electronic triage more accurately differentiates patients with respect to clinical outcomes compared with the emergency severity index. Ann Emerg Med. 2018; 71(5):565–574.e2.

16. Sterling NW, Patzer RE, Di M, Schrager JD. Prediction of emergency department patient disposition based on natural language processing of triage notes. Int J Med Inform. 2019; 129:184–188.

17. McKinney W. Data structures for statistical computing in python. In : Proceedings of the 9th Python in Science Conference; 2010 Jun 28-Jul 3; Austin, TX. p. 51–56.

18. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: machine learning in Python. J Mach Learn Res. 2011; 12:2825–2830.

19. Kim H. soynlp [Internet]. [place unknown]: github.com;2019. cited at 2019 Oct 16. Available from: https://github.com/lovit/soynlp.

20. Breiman L. Random forests. Mach Learn. 2001; 45(1):5–32.

21. Fernandez-Delgado M, Cernadas E, Barro S, Amorim D. Do we need hundreds of classifiers to solve real world classification problems? J Mach Learn Res. 2014; 15(1):3133–3181.

22. Chen T, Guestrin C. XGBoost: a scalable tree boosting system. In : Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 2016 Aug 13-17; San Francisco, CA. 785: 794.

23. Nielsen D. Tree boosting with XGBoost: why does XGBoost win “every” machine learning competition? [master's thesis]. Trondheim, Norway: Norwegian University of Science and Technology;2016.

24. Passalis N, Tefas A. Entropy optimized feature-based bag-of-words representation for information retrieval. IEEE Trans Knowl Data Eng. 2016; 28(7):1664–1677.

25. Sidorov G, Velasquez F, Stamatatos E, Gelbukh A, Chanona-Hernandez L. Syntactic n-grams as machine learning features for natural language processing. Expert Syst Appl. 2014; 41(3):853–860.

26. Park JB, Lee J, Kim YJ, Lee JH, Lim TH. Reliability of Korean Triage and Acuity Scale: interrater agreement between two experienced nurses by real-time triage and analysis of influencing factors to disagreement of triage levels. J Korean Med Sci. 2019; 34(28):e189.

27. Moon SH, Shim JL, Park KS, Park CS. Triage accuracy and causes of mistriage using the Korean Triage and Acuity Scale. PLoS One. 2019; 14(9):e0216972.

28. Levis T, Schwartz D, Bitan Y. Triage nurses decision-support application design. Proc Int Symp Hum Factors Ergon Healthc. 2018; 7(1):52–55.

29. Dehghani Soufi M, Samad-Soltani T, Shams Vahdati S, Rezaei-Hachesu P. Decision support system for triage management: a hybrid approach using rule-based reasoning and fuzzy logic. Int J Med Inform. 2018; 114:35–44.