Ann Lab Med.  2023 Sep;43(5):425-433. 10.3343/alm.2023.43.5.425.

A New Strategy for Evaluating the Quality of Laboratory Results for Big Data Research: Using External Quality Assessment Survey Data (2010–2020)

• Cho EJ
• Jeong TD
• Kim S
• Park HD
• Yun YM
• Chun S
• Min WK

• KMID: 2551990
• DOI: http://doi.org/10.3343/alm.2023.43.5.425

Abstract

Background
To ensure valid results of big data research in the medical field, the input laboratory results need to be of high quality. We aimed to establish a strategy for evaluating the quality of laboratory results suitable for big data research.
Methods
We used Korean Association of External Quality Assessment Service (KEQAS) data to retrospectively review multicenter data. Seven measurands were analyzed using commutable materials: HbA1c, creatinine (Cr), total cholesterol (TC), triglyceride (TG), alpha-fetoprotein (AFP), prostate-specific antigen (PSA), and cardiac troponin I (cTnI). These were classified into three groups based on their standardization or harmonization status. HbA1c, Cr, TC, TG, and AFP were analyzed with respect to peer group values. PSA and cTnI were analyzed in separate peer groups according to the calibrator type and manufacturer, respectively. The acceptance rate and absolute percentage bias at the medical decision level were calculated based on biological variation criteria.
Results
The acceptance rate (22.5%–100%) varied greatly among the test items, and the mean percentage biases were 0.6%–5.6%, 1.0%–9.6%, and 1.6%–11.3% for all items that satisfied optimum, desirable, and minimum criteria, respectively.
Conclusions
The acceptance rate of participants and their external quality assessment (EQA) results exhibited statistically significant differences according to the quality grade for each criterion. Even when they passed the EQA standards, the test results did not guarantee the quality requirements for big data. We suggest that the KEQAS classification can serve as a guide for building big data.

Keyword

Bias; Big data; Biological variation; Data quality; External quality assessment

Figure

• 

  Fig. 1 Overview of the approach used to categorize the data into groups and establish the target values. Abbreviations: KEQAS, Korean Association of External Quality Assessment Service; EQA, External Quality Assessment.

• 

  Fig. 2 Percentages of acceptable performances and sample concentrations ranging from approximately 6.0% to 7.0% obtained from 10 EQA samples for HbA1c according to different performance goals. Abbreviation: EQA, external quality assessment.

• 

  Fig. 3 Mean percentages of acceptable performances considering the participants’ EQA results according to different performance goals. Data categorized into three groups are shown for each test item. (A) Standardization of laboratory tests, including HbA1c, creatinine, total cholesterol, and triglyceride. (B) Harmonization category, including AFP and PSA classified according to the calibrator (Hybritech standard or WHO 96/670 IS). (C) Lack-of-harmonization category, including cardiac troponin I, according to six manufacturers (A–F). Abbreviations: EQA, external quality assessment; AFP, alpha-fetoprotein; PSA, prostate-specific antigen; IS, international standard.

• 

  Fig. 4 Box plot analysis of mean percentage bias for total samples for each test according to different performance goals. (A) HbA1c, (B) creatinine, (C) total cholesterol, (D) triglyceride, and (E) alpha-fetoprotein. The gray box plot shows the minimum, first quartile, median, third quartile, and maximum values. The blue line indicates the mean and the blue diamond the confidence interval of the data. Groups were compared using the Student–Newman–Keuls multiple-comparison test. **P≤0.001 and *P≤0.05.

• 

  Fig. 5 Box plot analysis of mean percentage bias for cardiac troponin I grouped into six manufacturers (A–F) according to different performance goals. Bars represent means, and error bars represent 95% confidence intervals. Groups were compared using the Student–Newman–Keuls multiple-comparison test. **P≤0.001 and *P≤0.05.

Reference

1. Wang L, Alexander CA. 2020; Big data analytics in medical engineering and healthcare: methods, advances and challenges. J Med Eng Technol. 44:267–83. DOI: 10.1080/03091902.2020.1769758. PMID: 32498594.

2. Rumsfeld JS, Joynt KE, Maddox TM. 2016; Big data analytics to improve cardiovascular care: promise and challenges. Nat Rev Cardiol. 13:350–9. DOI: 10.1038/nrcardio.2016.42. PMID: 27009423.

3. Hong L, Luo M, Wang R, Lu P, Lu W, Lu L. 2018; Big data in health care: applications and challenges. Data Inf Manag. 2:175–97. DOI: 10.2478/dim-2018-0014.

4. Mashoufi M, Ayatollahi H, Khorasani-Zavareh D. 2018; A review of data quality assessment in emergency medical services. Open Med Inform J. 12:19–32. DOI: 10.2174/1874431101812010019. PMID: 29997708. PMCID: PMC5997849.

5. Hallworth MJ. 2011; The '70% claim': what is the evidence base? Ann Clin Biochem. 48:487–8. DOI: 10.1258/acb.2011.011177. PMID: 22045648.

6. Division of Laboratory Systems (DLS), Centers for Disease Control and Prevention. https://www.cdc.gov/csels/dls/strengthening-clinical-labs.html. Updated on Jan 2023.

7. Kim S, Lee K, Park HD, Lee YW, Chun S, Min WK. 2021; Schemes and Performance Evaluation Criteria of Korean Association of External Quality Assessment (KEQAS) for Improving Laboratory Testing. Ann Lab Med. 41:230–9. DOI: 10.3343/alm.2021.41.2.230. PMID: 33063686. PMCID: PMC7591290.

8. John WG, Mosca A, Weykamp C, Goodall I. 2007; HbA1c standardisation: history, science and politics. Clin Biochem Rev. 28:163–8.

9. Nakamura M, Iso H, Kitamura A, Imano H, Kiyama M, Yokoyama S, et al. 2015; Total cholesterol performance of Abell-Levy-Brodie-Kendall reference measurement procedure: certification of Japanese in-vitro diagnostic assay manufacturers through CDC's cholesterol Reference Method Laboratory Network. Clin Chim Acta. 445:127–32. DOI: 10.1016/j.cca.2015.03.026. PMID: 25818239. PMCID: PMC4579524.

10. Nakamura M, Iso H, Kitamura A, Imano H, Noda H, Kiyama M, et al. 2016; Comparison between the triglycerides standardization of routine methods used in Japan and the chromotropic acid reference measurement procedure used by the CDC Lipid Standardization Programme. Ann Clin Biochem. 53:632–9. DOI: 10.1177/0004563215624461. PMID: 26680645. PMCID: PMC5695560.

11. Myers GL. 2008; Standardization of serum creatinine measurement: theory and practice. Scand J Clin Lab Invest Suppl. 241:57–63. DOI: 10.1080/00365510802149887. PMID: 18569966.

12. Kim S, Cho EJ, Jeong TD, Park HD, Yun YM, Lee K, et al. 2023; Proposed model for evaluating real-world laboratory results for big data research. Ann Lab Med. 43:104–7. DOI: 10.3343/alm.2023.43.1.104. PMID: 36045065. PMCID: PMC9467825.

13. Kim JH, Cho Y, Lee SG, Yun YM. 2019; Report of Korean association of external quality assessment service on the accuracy-based lipid proficiency testing (2016-2018). J Lab Med Qual Assur. 41:121–9. DOI: 10.15263/jlmqa.2019.41.3.121.

14. International Consortium for Harmonization of Clinical Laboratory Results (ICHCLR). www.harmonization.net. Updated on Nov 2021.

15. Ferraro S, Panzeri A, Braga F, Panteghini M. 2019; Serum α-fetoprotein in pediatric oncology: not a children's tale. Clin Chem Lab Med. 57:783–97. DOI: 10.1515/cclm-2018-0803. PMID: 30367785.

16. Ferraro S, Bussetti M, Rizzardi S, Braga F, Panteghini M. 2021; Verification of harmonization of serum total and free prostate-specific antigen (PSA) measurements and implications for medical decisions. Clin Chem. 67:543–53. DOI: 10.1093/clinchem/hvaa268. PMID: 33674839.

17. American Diabetes Association 6. 2021; Glycemic targets: standards of medical care in Diabetes-2021. Diabetes Care. 44:S73–84. DOI: 10.2337/dc21-S006. PMID: 33298417.

18. Park EY, Kim TY. 2010; Where are cut-off values of serum creatinine in the setting of chronic kidney disease? Kidney Int. 77:645–6. DOI: 10.1038/ki.2009.529. PMID: 20224585.

19. Grundy SM, Cleeman JI, Merz CN, Brewer HB Jr, Clark LT, Hunninghake DB, et al. 2004; Implications of recent clinical trials for the National Cholesterol Education Program Adult Treatment Panel III guidelines. Circulation. 110:227–39. DOI: 10.1161/01.CIR.0000133317.49796.0E. PMID: 15249516.

20. Gambarin-Gelwan M, Wolf DC, Shapiro R, Schwartz ME, Min AD. 2000; Sensitivity of commonly available screening tests in detecting hepatocellular carcinoma in cirrhotic patients undergoing liver transplantation. Am J Gastroenterol. 95:1535–8. DOI: 10.1111/j.1572-0241.2000.02091.x. PMID: 10894592.

21. Trevisani F, D'Intino PE, Morselli-Labate AM, Mazzella G, Accogli E, Caraceni P, et al. 2001; Serum alpha-fetoprotein for diagnosis of hepatocellular carcinoma in patients with chronic liver disease: influence of HBsAg and anti-HCV status. J Hepatol. 34:570–5. DOI: 10.1016/S0168-8278(00)00053-2. PMID: 11394657.

22. Carter HB, Albertsen PC, Barry MJ, Etzioni R, Freedland SJ, Greene KL, et al. 2013; Early detection of prostate cancer: AUA Guideline. J Urol. 190:419–26. DOI: 10.1016/j.juro.2013.04.119. PMID: 23659877. PMCID: PMC4020420.

23. Contemporary cardiac troponin I and T assay analytical characteristics designated by manufacturer IFCC committee on clinical applications of cardiac bio-markers (C-CB) v052022. https://ifcc.org/ifcc-education-division/emd-committees/committee-on-clinical-applications-of-cardiac-bio-markers-c-cb/biomarkers-reference-tables/. Updated on Jan 2023.

24. Aarsand AK, Fernandez-Calle P, Webster C, Coskun A, Gonzales-Lao E, Diaz-Garzon J, et al. The EFLM Biological Variation Database. https://biologicalvariation.eu/. Updated on Nov 2021.

25. Weykamp CW, Mosca A, Gillery P, Panteghini M. 2011; The analytical goals for hemoglobin A(1c) measurement in IFCC units and National Glycohemoglobin Standardization Program Units are different. Clin Chem. 57:1204–6. DOI: 10.1373/clinchem.2011.162719. PMID: 21571810.

26. Myers GL, Miller WG, Coresh J, Fleming J, Greenberg N, Greene T, et al. 2006; Recommendations for improving serum creatinine measurement: a report from the Laboratory Working Group of the National Kidney Disease Education Program. Clin Chem. 52:5–18. DOI: 10.1373/clinchem.2005.0525144. PMID: 16332993.

27. Warnick GR, Kimberly MM, Waymack PP, Leary ET, Myers GL. 2008; Standardization of measurements for cholesterol, triglycerides, and major lipoproteins. Lab Med. 39:481–90. DOI: 10.1309/6UL9RHJH1JFFU4PY.

28. Hripcsak G, Knirsch C, Zhou L, Wilcox A, Melton G. 2011; Bias associated with mining electronic health records. J Biomed Discov Collab. 6:48–52. DOI: 10.5210/disco.v6i0.3581. PMID: 21647858. PMCID: PMC3149555.

29. Weiskopf NG, Hripcsak G, Swaminathan S, Weng C. 2013; Defining and measuring completeness of electronic health records for secondary use. J Biomed Inform. 46:830–6. DOI: 10.1016/j.jbi.2013.06.010. PMID: 23820016. PMCID: PMC3810243.

30. Marcovina SM, Gaur VP, Albers JJ. 1994; Biological variability of cholesterol, triglyceride, low- and high-density lipoprotein cholesterol, lipoprotein(a), and apolipoproteins A-I and B. Clin Chem. 40:574–8. DOI: 10.1093/clinchem/40.4.574. PMID: 8149613.

31. Vignati G, Giovanelli L. 2007; Standardization of PSA measures: a reappraisal and an experience with WHO calibration of Beckman Coulter Access Hybritech total and free PSA. Int J Biol Markers. 22:295–301. DOI: 10.1177/172460080702200409. PMID: 18161661.

32. Stephan C, Bangma C, Vignati G, Bartsch G, Lein M, Jung K, et al. 2009; 20-25% lower concentrations of total and free prostate-specific antigen (PSA) after calibration of PSA assays to the WHO reference materials-analysis of 1098 patients in four centers. Int J Biol Markers. 24:65–9. DOI: 10.5301/JBM.2009.1349. PMID: 19634108.

33. Marques-Garcia F, Boned B, González-Lao E, Braga F, Carobene A, Coskun A, et al. 2022; Critical review and meta-analysis of biological variation estimates for tumor markers. Clin Chem Lab Med. 60:494–504. DOI: 10.1515/cclm-2021-0725. PMID: 35143717.

34. Carobene A, Guerra E, Locatelli M, Cucchiara V, Briganti A, Aarsand AK, et al. 2018; Biological variation estimates for prostate specific antigen from the European Biological Variation Study; consequences for diagnosis and monitoring of prostate cancer. Clin Chim Acta. 486:185–91. DOI: 10.1016/j.cca.2018.07.043. PMID: 30063887.

35. Christenson RH, Jacobs E, Uettwiller-Geiger D, Estey MP, Lewandrowski K, Koshy TI, et al. 2017; Comparison of 13 commercially available cardiac troponin assays in a multicenter North American study. J Appl Lab Med. 2:134. DOI: 10.1373/jalm.2017.023903. PMID: 33636962.

36. Kim HS, Kim DJ, Yoon KH. 2019; Medical big data is not yet available: why we need realism rather than exaggeration. Endocrinol Metab (Seoul). 34:349–54. DOI: 10.3803/EnM.2019.34.4.349. PMID: 31884734. PMCID: PMC6935779.

37. Dash S, Shakyawar SK, Sharma M, Kaushik S. 2019; Big data in healthcare: management, analysis and future prospects. J Big Data. 6:54. DOI: 10.1186/s40537-019-0217-0.

38. Shi X, Prins C, Van Pottelbergh G, Mamouris P, Vaes B, De Moor B. 2021; An automated data cleaning method for electronic health records by incorporating clinical knowledge. BMC Med Inform Decis Mak. 21:267. DOI: 10.1186/s12911-021-01630-7. PMID: 34535146. PMCID: PMC8449435.

39. Vesper HW, Myers GL, Miller WG. 2016; Current practices and challenges in the standardization and harmonization of clinical laboratory tests. Am J Clin Nutr. 104(Suppl 3):907S–12S. DOI: 10.3945/ajcn.115.110387. PMID: 27534625. PMCID: PMC5004491.

40. Panteghini M. 2012; Implementation of standardization in clinical practice: not always an easy task. Clin Chem Lab Med. 50:1237–41. DOI: 10.1515/cclm.2011.791. PMID: 22850055.