Korean J Neurotrauma.  2019 Oct;15(2):88-94. 10.13004/kjnt.2019.15.e17.

Application of Deep Learning System into the Development of Communication Device for Quadriplegic Patient

• Lee JH
• Kang T
• Choi BK
• Han IH
• Kim BC
• Ro JH

• KMID: 2461112
• DOI: http://doi.org/10.13004/kjnt.2019.15.e17

Abstract

OBJECTIVE
In general, quadriplegic patients use their voices to call the caregiver. However, severe quadriplegic patients are in a state of tracheostomy, and cannot generate a voice. These patients require other communication tools to call caregivers. Recently, monitoring of eye status using artificial intelligence (AI) has been widely used in various fields. We made eye status monitoring system using deep learning, and developed a communication system for quadriplegic patients can call the caregiver.
METHODS
The communication system consists of 3 programs. The first program was developed for automatic capturing of eye images from the face using a webcam. It continuously captured and stored 15 eye images per second. Secondly, the captured eye images were evaluated for open or closed status by deep learning, which is a type of AI. Google TensorFlow was used as a machine learning tool or library for convolutional neural network. A total of 18,000 images were used to train deep learning system. Finally, the program was developed to utter a sound when the left eye was closed for 3 seconds.
RESULTS
The test accuracy of eye status was 98.7%. In practice, when the quadriplegic patient looked at the webcam and closed his left eye for 3 seconds, the sound for calling a caregiver was generated.
CONCLUSION
Our eye status detection software using AI is very accurate, and the calling system for the quadriplegic patient was satisfactory.

Keyword

Eye; Artificial intelligence; Unsupervised machine learning; Quadriplegia; Caregiver

MeSH Terms

Artificial Intelligence
Caregivers
Humans
Learning*
Machine Learning
Quadriplegia
Tracheostomy
Unsupervised Machine Learning
Voice

Figure

• 

  FIGURE 1 Both eyes are captured by the webcam.

• 

  FIGURE 2 Captured eye image. The size is 28×28 pixels, a total of 784 pixels.

• 

  FIGURE 3 Software coded using Google Tensorflow. The upper image is the algorithm related to sending and receiving eye images to the artificial intelligence server. The lower image is the algorithm related to generating a sound when the left eye was closed continuously for 3 seconds once the right eye was open.

• 

  FIGURE 4 Practical applications in a quadriplegic patient. (A) Actual setting for the quadriplegic patient. (B) Window of the software. Click ‘btn 1’ button to recognize both eyes. Click ‘log start’ button to recognize eye status, and call the caregiver. When the left eye was closed for 3 seconds, a sound was generated to call the caregiver. The ‘log end’ button was used to stop distinguishing eye status.

Reference

1. Chang WD, Lim JH, Im CH. An unsupervised eye blink artifact detection method for real-time electroencephalogram processing. Physiol Meas. 2016; 37:401–417. PMID: 26888113.

2. Choi JS, Bang JW, Heo H, Park KR. Evaluation of fear using nonintrusive measurement of multimodal sensors. Sensors (Basel). 2015; 15:17507–17533. PMID: 26205268.

3. Dominguez Veiga JJ, O'Reilly M, Whelan D, Caulfield B, Ward TE. Feature-free activity classification of inertial sensor data with machine vision techniques: method, development, and evaluation. JMIR Mhealth Uhealth. 2017; 5:e115. PMID: 28778851.

4. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542:115–118. PMID: 28117445.

5. Feng R, Badgeley M, Mocco J, Oermann EK. Deep learning guided stroke management: a review of clinical applications. J Neurointerv Surg. 2018; 10:358–362. PMID: 28954825.

6. Harrop JS, Sharan AD, Scheid EH Jr, Vaccaro AR, Przybylski GJ. Tracheostomy placement in patients with complete cervical spinal cord injuries: American Spinal Injury Association grade A. J Neurosurg. 2004; 100:20–23. PMID: 14748569.

7. Hirasawa T, Aoyama K, Tanimoto T, Ishihara S, Shichijo S, Ozawa T, et al. Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer. 2018; 21:653–660. PMID: 29335825.

8. Kim KW, Hong HG, Nam GP, Park KR. A study of deep CNN-based classification of open and closed eyes using a visible light camera sensor. Sensors (Basel). 2017; 17:E1534. PMID: 28665361.

9. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012; 25:1097–1105.

10. Lugo Z, Pellas F, Blandin V, Laureys S, Gosseries O. Assessment of needs, psychological impact and quality of life in families of patients with locked-in syndrome. Brain Inj. 2017; 31:1590–1596. PMID: 28837360.

11. Nakayama Y, Shimizu T, Mochizuki Y, Hayashi K, Matsuda C, Nagao M, et al. Predictors of impaired communication in amyotrophic lateral sclerosis patients with tracheostomy-invasive ventilation. Amyotroph Lateral Scler Frontotemporal Degener. 2015; 17:38–46. PMID: 26121169.

12. Olczak J, Fahlberg N, Maki A, Razavian AS, Jilert A, Stark A, et al. Artificial intelligence for analyzing orthopedic trauma radiographs. Acta Orthop. 2017; 88:581–586. PMID: 28681679.

13. Tafreshi M, Fotouhi AM. A fast and accurate algorithm for eye opening or closing detection based on local maximum vertical derivative pattern. Turk J Electr Eng Comput Sci. 2016; 24:5124–5134.

14. van Middendorp JJ, Watkins F, Park C, Landymore H. Eye-tracking computer systems for inpatients with tetraplegia: findings from a feasibility study. Spinal Cord. 2015; 53:221–225. PMID: 25448188.

15. Wezel J, Garpebring A, Webb AG, van Osch MJ, Beenakker JM. Automated eye blink detection and correction method for clinical MR eye imaging. Magn Reson Med. 2017; 78:165–171. PMID: 27476861.