J Pathol Transl Med.  2019 Jan;53(1):1-12. 10.4132/jptm.2018.12.16.

Artificial Intelligence in Pathology

• Chang HY
• Jung CK
• Woo JI
• Lee S
• Cho J
• Kim SW
• Kwak TY

• KMID: 2437573
• DOI: http://doi.org/10.4132/jptm.2018.12.16

Abstract

As in other domains, artificial intelligence is becoming increasingly important in medicine. In particular, deep learning-based pattern recognition methods can advance the field of pathology by incorporating clinical, radiologic, and genomic data to accurately diagnose diseases and predict patient prognoses. In this review, we present an overview of artificial intelligence, the brief history of artificial intelligence in the medical domain, recent advances in artificial intelligence applied to pathology, and future prospects of pathology driven by artificial intelligence.

Keyword

Artificial intelligence; Deep learning; Pathology; Image analysis

MeSH Terms

Artificial Intelligence*
Humans
Pathology*
Prognosis

Figure

• 

  Fig. 1. A simplified modern convolutional neural network (CNN) architecture example. In contrast to the classic CNN comprising only a cascade of convolution layers and pooling layers followed by a few fully connected layers, this example has various other concepts like branching from the max pooling layer to several (1×1, 3×3, 5 × 5) convolution layers as well as the average pooling layer, merging by concatenation from two (1×1, 5×5) convolution layers and the average pooling layer, and residual addition of max pooling layer output to the output of its succeeding (3×3) convolution layer.

• 

  Fig. 2. A typical recurrent layer example. In receiving a new input xt at time t, hidden state ht is updated based on xt and the previous state ht-1 first, then output yt is generated based on ht. At training time, parameters like U, V, W, bh, and by are trained to accurately generate yt for every time t.

• 

  Fig. 3. An example workflow for two-stage pathology artificial intelligence. Training phase: from the collected pathology images, a proper amount of annotation data is constructed (a). Image patch sets of balanced size are used in the training of patch-level convolutional neural network (CNN). After the patch-level CNN is trained sufficiently, heatmaps are generated for another set of pathology images using that CNN, from where the features are extracted for the decision forest like image-level machine learning (ML) model training (b). Inference phase: patch-level CNN runs on every single patch in the input pathology to generate a heatmap (first stage). Features are then extracted as in the training phase, and fed into the image-level ML model to determine the image-level result (second stage).

Reference

1. McCorduck P. Machines who think: a personal inquiry into the history and prospects of artificial intelligence. Natick: A.K. Peters;2004.

2. Turing AM. I. Computing machinery and intelligence. Mind. 1950; 59:433–60.

3. Searle JR. Minds, brains, and programs. Behav Brain Sci. 1980; 3:417–24.

4. Russell SJ, Norvig P. Artificial intelligence: a modern approach. Upper Saddle River: Prentice Hall;2003.

5. Artificial intelligence [Internet]. Wikipedia;2018. [cited 2018 Dec 9]. Available from: https://en.wikipedia.org/wiki/Artificial_intelligence.

6. Mortensen TL, Watt DL, Leistritz FL. Loan default prediction using logistic regression and a loan pricing model. Report No. 119549 [Internet]. Fargo: North Dakota State University;1988. [cited 2018 Dec 7]. Available from: https://ideas.repec.org/p/ags/nddmrs/119549.html.

7. Graham P. Better Bayesian filtering [Internet]. PAUL GRAHAM;2003. [cited 2018 Nov 22]. Available from: http://www.paulgraham.com/better.html.

8. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. In: Pereira F, Burges CJ, Bottou L, Weinberger KQ, eds. Advances in neural information processing systems 25. Red Hook: Curran Associates, Inc;2012. p. 1097–105.

9. Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning. Nature. 2015; 518:529–33.

10. Silver D, Schrittwieser J, Simonyan K, et al. Mastering the game of Go without human knowledge. Nature. 2017; 550:354–9.

11. Hannun A, Case C, Casper J. Deep speech: scaling up end-to-end speech recognition [Internet]. Ithaca: arXiv, Cornell University;2014. [cited 2018 Nov 22]. Available from: http://arxiv.org/abs/1412.5567.

12. Luong MT, Pham H, Manning CD. Effective approaches to attention-based neural machine translation. In : In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing; 2015 Sep 17-21; Lisbon, Portugal. Stroudsburg. Association for Computational Linguistics. 2015. p. 1412–21.

13. Wu Y, Schuster M, Chen Z. Google’s neural machine translation system: bridging the gap between human and machine translation [Internet]. Ithaca: arXiv, Cornell University;2016. [cited 2018 Nov 22]. Available from: http://arxiv.org/abs/1609.08144.

14. Antol S, Agrawal A, Lu J, et al. VQA: visual question answering. In : In: Proceedings of the IEEE International Conference on Computer Vision; 2015 Dec 7-13; Santiago, Chile. Washington, DC. IEEE Computer Society. 2015. p. 2425–33.

15. Kim JH, Lee SW, Kwak D, et al. Multimodal residual learning for visual QA. In : Lee DD, von Luxburg U, Garnett R, editors. Advances in neural information processing systems 29. Red Hook: NY Curran Associates Inc;2016. p. 361–9.

16. LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proc IEEE. 1998; 86:2278–324.

17. Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015 Jun 7-12, Boston, MA, USA. Silver Spring: IEEE Computer Society Press;2015. p. 1–9.

18. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature. 2015; 521:436–44.

19. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997; 9:1735–80.

20. Weizenbaum J. ELIZA: a computer program for the study of natural language communication between man and machine. Commun ACM. 1966; 9:36–45.

21. Shortliffe EH. Mycin: a knowledge-based computer program applied to infectious diseases. In: Proceedings of the Annual Symposium on Computer Application in Medical Care, 1977 Oct 3-5, Washington, DC, USA. New York: Institute of Electrical and Electronics Engineers;1977. p. 66–9.

22. Heckerman DE, Horvitz EJ, Nathwani BN. Toward normative expert systems: Part I. The Pathfinder project. Methods Inf Med. 1992; 31:90–105.

23. Heckerman DE, Nathwani BN. Toward normative expert systems: Part II. Probability-based representations for efficient knowledge acquisition and inference. Methods Inf Med. 1992; 31:106–16.

24. Vyborny CJ, Giger ML. Computer vision and artificial intelligence in mammography. AJR Am J Roentgenol. 1994; 162:699–708.

25. Kononenko I. Machine learning for medical diagnosis: history, state of the art and perspective. Artif Intell Med. 2001; 23:89–109.

26. Baker JA, Rosen EL, Lo JY, Gimenez EI, Walsh R, Soo MS. Computeraided detection (CAD) in screening mammography: sensitivity of commercial CAD systems for detecting architectural distortion. AJR Am J Roentgenol. 2003; 181:1083–8.

27. Gulshan V, Peng L, Coram M, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016; 316:2402–10.

28. Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image analysis. Med Image Anal. 2017; 42:60–88.

29. Kohli M, Prevedello LM, Filice RW, Geis JR. Implementing machine learning in radiology practice and research. AJR Am J Roentgenol. 2017; 208:754–60.

30. Shaikhina T, Khovanova NA. Handling limited datasets with neural networks in medical applications: a small-data approach. Artif Intell Med. 2017; 75:51–63.

31. Angermueller C, Parnamaa T, Parts L, Stegle O. Deep learning for computational biology. Mol Syst Biol. 2016; 12:878.

32. Torkamani A, Andersen KG, Steinhubl SR, Topol EJ. High-definition medicine. Cell. 2017; 170:828–43.

33. Wainberg M, Merico D, Delong A, Frey BJ. Deep learning in biomedicine. Nat Biotechnol. 2018; 36:829–38.

34. Xiong HY, Alipanahi B, Lee LJ, et al. RNA splicing: the human splicing code reveals new insights into the genetic determinants of disease. Science. 2015; 347:1254806.

35. Poplin R, Chang PC, Alexander D, et al. A universal SNP and smallindel variant caller using deep neural networks. Nat Biotechnol. 2018; 36:983–7.

36. Rajkomar A, Oren E, Chen K, et al. Scalable and accurate deep learning with electronic health records. NPJ Digit Med. 2018; 1:18.

37. Fernandes K, Chicco D, Cardoso JS, Fernandes J. Supervised deep learning embeddings for the prediction of cervical cancer diagnosis. PeerJ Comput Sci. 2018; 4:e154.

38. Ye JJ. Artificial intelligence for pathologists is not near: it is here: description of a prototype that can transform how we practice pathology tomorrow. Arch Pathol Lab Med. 2015; 139:929–35.

39. Beck JR, Salem DN, Estes NA, Pauker SG. A computer-based Markov decision analysis of the management of symptomatic bifascicular block: the threshold probability for pacing. J Am Coll Cardiol. 1987; 9:920–35.

40. Schaefer AJ, Bailey MD, Shechter SM, Roberts MS. Modeling medical treatment using Markov decision processes. In: Brandeau ML, Sainfort F, Pierskalla WP, eds. Operations research and health care: a handbook of methods and applications. Boston: Kluwer Academic Publisher;2004. p. 593–612.

41. Alagoz O, Hsu H, Schaefer AJ, Roberts MS. Markov decision processes: a tool for sequential decision making under uncertainty. Med Decis Making. 2010; 30:474–83.

42. Harbias A, Salmo E, Crump A. Implications of observer variation in Gleason scoring of prostate cancer on clinical management: a collaborative audit. Gulf J Oncolog. 2017; 1:41–5.

43. Ozkan TA, Eruyar AT, Cebeci OO, Memik O, Ozcan L, Kuskonmaz I. Interobserver variability in Gleason histological grading of prostate cancer. Scand J Urol. 2016; 50:420–4.

44. Janowczyk A, Madabhushi A. Deep learning for digital pathology image analysis: a comprehensive tutorial with selected use cases. J Pathol Inform. 2016; 7:29.

45. Komura D, Ishikawa S. Machine learning methods for histopathological image analysis. Comput Struct Biotechnol J. 2018; 16:34–42.

46. Garud H, Karri SP, Sheet D, et al. High-magnification multi-views based classification of breast fine needle aspiration cytology cell samples using fusion of decisions from deep convolutional networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017 Jul 21-26, Honolulu, HI, USA. New York: Institute of Electrical and Electronics Engineers;2017. p. 828–33.

47. Li Y, Ping W. Cancer metastasis detection with neural conditional random field [Internet]. Ithaca: arXiv, Cornell University;2018. [cited 2018 Nov 22]. Available from: http://arxiv.org/abs/1806.07064.

48. Rannen Triki A, Blaschko MB, Jung YM, et al. Intraoperative margin assessment of human breast tissue in optical coherence tomography images using deep neural networks. Comput Med Imaging Graph. 2018; 69:21–32.

49. Ehteshami Bejnordi B, Mullooly M, Pfeiffer RM, et al. Using deep convolutional neural networks to identify and classify tumor-associated stroma in diagnostic breast biopsies. Mod Pathol. 2018; 31:1502–12.

50. Litjens G, Sánchez CI, Timofeeva N, et al. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Sci Rep. 2016; 6:26286.

51. Cires¸an DC, Giusti A, Gambardella LM, Schmidhuber J. Mitosis detection in breast cancer histology images with deep neural networks. Med Image Comput Comput Assist Interv. 2013; 16:411–8.

52. Teramoto A, Tsukamoto T, Kiriyama Y, Fujita H. Automated classification of lung cancer types from cytological images using deep convolutional neural networks. Biomed Res Int. 2017; 2017:4067832.

53. Yu KH, Zhang C, Berry GJ, et al. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nat Commun. 2016; 7:12474.

54. Coudray N, Ocampo PS, Sakellaropoulos T, et al. Classification and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nat Med. 2018; 24:1559–67.

55. Campanella G, Silva VW, Fuchs TJ. Terabyte-scale deep multiple instance learning for classification and localization in pathology [Internet]. Ithaca: arXiv, Cornell University;2018. [cited 2018 Nov 22]. Available from: http://arxiv.org/abs/1805.06983.

56. Arvaniti E, Fricker KS, Moret M, et al. Automated Gleason grading of prostate cancer tissue microarrays via deep learning. Sci Rep. 2018; 8:12054.

57. Zhou N, Fedorov A, Fennessy F, Kikinis R, Gao Y. Large scale digital prostate pathology image analysis combining feature extraction and deep neural network [Internet]. Ithaca: arXiv, Cornell University;2017. [cited 2018 Nov 22]. Available from: http://arxiv.org/abs/1705.02678.

58. Nagpal K, Foote D, Liu Y, et al. Development and validation of a deep learning algorithm for improving Gleason scoring of prostate cancer [Internet]. Ithaca: arXiv, Cornell University;2018. [cited 2018 Nov 22]. Available from: http://arxiv.org/abs/1811.06497.

59. Ertosun MG, Rubin DL. Automated grading of gliomas using deep learning in digital pathology images: a modular approach with ensemble of convolutional neural networks. AMIA Annu Symp Proc. 2015; 2015:1899–908.

60. Mobadersany P, Yousefi S, Amgad M, et al. Predicting cancer outcomes from histology and genomics using convolutional networks. Proc Natl Acad Sci U S A. 2018; 115:E2970–E9.

61. Wu M, Yan C, Liu H, Liu Q. Automatic classification of ovarian cancer types from cytological images using deep convolutional neural networks. Biosci Rep. 2018; 38:BSR20180289.

62. Zhang L, Lu L, Nogues I, Summers RM, Liu S, Yao J. DeepPap: deep convolutional networks for cervical cell classification. IEEE J Biomed Health Inform. 2017; 21:1633–43.

63. Xu M, Papageorgiou DP, Abidi SZ, Dao M, Zhao H, Karniadakis GE. A deep convolutional neural network for classification of red blood cells in sickle cell anemia. PLoS Comput Biol. 2017; 13:e1005746.

64. Meier A, Nekolla K, Earle S, et al. End-to-end learning to predict survival in patients with gastric cancer using convolutional neural networks. Ann Oncol. 2018; 29(Suppl 8):mdy269.075.

65. Xie W, Noble JA, Zisserman A. Microscopy cell counting and detection with fully convolutional regression networks. Comput Methods Biomech Biomed Eng Imaging Vis. 2016; 6:283–92.

66. Tuominen VJ, Ruotoistenmaki S, Viitanen A, Jumppanen M, Isola J. ImmunoRatio: a publicly available web application for quantitative image analysis of estrogen receptor (ER), progesterone receptor (PR), and Ki-67. Breast Cancer Res. 2010; 12:R56.

67. Meijering E. Cell segmentation: 50 years down the road [life sciences]. IEEE Signal Process Mag. 2012; 29:140–5.

68. Ruifrok AC, Johnston DA. Quantification of histochemical staining by color deconvolution. Anal Quant Cytol Histol. 2001; 23:291–9.

69. Otsu N. A threshold selection method from gray-level histograms. IEEE Trans Syst Man Cybern. 1979; 9:62–6.

70. Zhang L, Sonka M, Lu L, Summers RM, Yao J. Combining fully convolutional networks and graph-based approach for automated segmentation of cervical cell nuclei. In: 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017), 2017 Apr 18-21, Melbourne, VIC, Australia. New York: Institute of Electrical and Electronics Engineers;2017. p. 406–9.

71. Chen H, Qi X, Yu L, Heng PA. DCAN: deep contour-aware networks for accurate gland segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016 Jun 27-30, Las Vegas, NV, USA. New York: Institute of Electrical and Electronics Engineers;2016. p. 2487–96.

72. CAMELYON16 Consortium. CAMELYON16. CAMELYON16 ISBI challenge on cancer metastasis detection in lymph node, 2015 [Internet]. Grand-Challenges;2016. [cited 2018 Nov 22]. Available from: https://camelyon16.grand-challenge.org/.

73. The Cancer Genome Atlas [Internet]. Bethesda: The Cancer Genome Atlas, National Cancer Institute;2011. [cited 2018 Nov 22]. Available from: https://cancergenome.nih.gov/.