J Korean Med Sci.  2014 Jun;29(6):771-775. 10.3346/jkms.2014.29.6.771.

Improving the Reliability of Clinical Practice Guideline Appraisals: Effects of the Korean AGREE II Scoring Guide

• Oh MK
• Jo H
• Lee YK

• KMID: 1796937
• DOI: http://doi.org/10.3346/jkms.2014.29.6.771

Abstract

The Korean translated Appraisal of Guidelines for Research and Evaluation II (Korean AGREE II) instrument was distributed into Korean medical societies in 2011. However, inter-rater disagreement issues still exist. The Korean AGREE II scoring guide was therefore developed to reduce inter-rater differences. This study examines the effects of the Korean AGREE II scoring guide to reduce inter-rater differences. Appraisers were randomly assigned to two groups (Scoring Guide group and Non-Scoring Guide group). The Korean AGREE II instrument was provided to both groups. However, the scoring guide was offered to Scoring Guide group only. Total 14 appraisers were participated and each guideline was assessed by 8 appraisers. To evaluate the reliability of the Korean AGREE II scoring guide, correlation of scores among appraisers and domain-specific intra-class correlation (ICC) were compared. Most scores of two groups were comparable. Scoring Guide group showed higher reliability at all guidelines. They showed higher correlation among appraisers and higher ICC values at almost all domains. The scoring guide reduces the inter-rater disagreement and improves the overall reliability of the Korean-AGREE II instrument.

Keyword

Clinical Practice Guideline; Observer Variation; Reproducibility of Results

MeSH Terms

Asian Continental Ancestry Group
Evidence-Based Medicine
Humans
*Practice Guidelines as Topic
Reproducibility of Results
Republic of Korea
Societies, Medical
Translating

Figure

• 

  Fig. 1 Distribution of Korean AGREE II domain scores according to the use of Scoring Guide for Cancer pain management guideline (A) and GERD guideline (B). Black boxs are Scoring guide group and white boxs are Non-Scoring Guide group. The top and bottom of the box indicates the 75th (Q3) and 25th percentile (Q1), respectively, and the horizontal line in the box means the 50th percentile (the median). The upper and lower ends of the whisker represent Q3+1.5×(interquartile range), and Q1-1.5×(interquartile range), respectively.

Reference

1. Shin YS, Kim YI. Health policy and management. Seoul: Seoul National University Press;2013.

2. The Korean Society for Preventive Medicine. Preventive medicine and public health. Seoul: Gyechuk munwhasa;2010.

3. Korean Medical Guideline Information Center. accessed on 9 July 2013. Available at http://www.guideline.or.kr/contents/index.php?code=015.

4. Darling G. The impact of clinical practice guidelines and clinical trials on treatment decisions. Surg Oncol. 2002; 11:255–262.

5. Legido-Quigley H, Panteli D, Brusamento S, Knai C, Saliba V, Turk E, Solé M, Augustin U, Car J, McKee M, et al. Clinical guidelines in the European Union: mapping the regulatory basis, development, quality control, implementation and evaluation across member states. Health Policy. 2012; 107:146–156.

6. AGREE Collaboration. Development and validation of an international appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care. 2003; 12:18–23.

7. Lee YK, Shin ES, Shim JY, Min KJ, Kim JM, Lee SH. the Executive Committee for CPGs. the Korean Academy of Medical Sciences. Developing a scoring guide for the Appraisal of Guidelines for Research and Evaluation II instrument in Korea: a modified Delphi consensus process. J Korean Med Sci. 2013; 28:190–194.

8. Sabharwal S, Patel V, Nijjer SS, Kirresh A, Darzi A, Chambers JC, Malik I, Kooner JS, Athanasiou T. Guidelines in cardiac clinical practice: evaluation of their methodological quality using the AGREE II instrument. J R Soc Med. 2013; 106:315–322.

9. Wijkstra J, Schubart CD, Nolen WA. Treatment of unipolar psychotic depression: the use of evidence in practice guidelines. World J Biol Psychiatry. 2009; 10:409–415.

10. Nagy E, Watine J, Bunting PS, Onody R, Oosterhuis WP, Rogic D, Sandberg S, Boda K, Horvath AR. IFCC Task Force on the Global Campaign for Diabetes Mellitus. Do guidelines for the diagnosis and monitoring of diabetes mellitus fulfill the criteria of evidence-based guideline development? Clin Chem. 2008; 54:1872–1882.

11. Holmer HK, Ogden LA, Burda BU, Norris SL. Quality of clinical practice guidelines for glycemic control in type 2 diabetes mellitus. PloS One. 2013; 8:e58625.

12. Van der Wees PJ, Hendriks EJ, Custers JW, Burgers JS, Dekker J, de Bie RA. Comparison of international guideline programs to evaluate and update the Dutch program for clinical guideline development in physical therapy. BMC Health Serv Res. 2007; 7:191.

13. Ansari S, Rashidian A. Guidelines for guidelines: are they up to the task? a comparative assessment of clinical practice guideline development handbooks. PloS One. 2012; 7:e49864.

14. Jo MW, Lee JY, Kim NS, Kim SY, Sheen S, Kim SH, Lee SI. Assessment of the quality of clinical practice guidelines in Korea using the AGREE Instrument. J Korean Med Sci. 2013; 28:357–365.

15. Esandi ME, Ortiz Z, Chapman E, Dieguez MG, Mejía R, Bernztein R. Production and quality of clinical practice guidelines in Argentina (1994-2004): a cross-sectional study. Implement Sci. 2008; 3:43.

16. Tremblay MS, Warburton DE, Janssen I, Paterson DH, Latimer AE, Rhodes RE, Kho ME, Hicks A, LeBlanc AG, Zehr L, et al. New Canadian physical activity guidelines. Appl Physiol Nutr Metab. 2011; 36:36–46.

17. Rossignol M, Poitras S, Dionne C, Tousignant M, Truchon M, Arsenault B, Allard P, Coté M, Neveu A. An interdisciplinary guideline development process: the Clinic on Low-back pain in Interdisciplinary Practice (CLIP) low-back pain guidelines. Implement Sci. 2007; 2:36.

18. Yan J, Min J, Zhou B. Diagnosis of pheochromocytoma: a clinical practice guideline appraisal using AGREE II instrument. J Eval Clin Pract. 2013; 19:626–632.

19. MacDermid JC, Brooks D, Solway S, Switzer-McIntyre S, Brosseau L, Graham ID. Reliability and validity of the AGREE instrument used by physical therapists in assessment of clinical practice guidelines. BMC Health Serv Res. 2005; 5:18.

20. Burgers JS, Cluzeau FA, Hanna SE, Hunt C, Grol R. Characteristics of high-quality guidelines: evaluation of 86 clinical guidelines developed in ten European countries and Canada. Int J Technol Assess Health Care. 2003; 19:148–157.

21. Lee WY. Review on the patient and public involvement in health technology appraisals at NICE. J Crit Soc Welfare. 2012; 34:47–75.

22. Kwon SM, You MS, Oh JH, Kim SJ, Jeon BY. Public participation in healthcare decision making: experience of citizen council for health insurance. Korean J Health Policy Adm. 2012; 22:467–496.

23. Kim YK, Lee SH, Seo JH, Kim JH, Kim SD, Kim GK. A comprehensive model of factors affecting adoption of clinical practice guidelines in Korea. J Korean Med Sci. 2010; 25:1568–1573.

24. Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006; 26:13–24.

25. Grimshaw JM, Thomas RE, MacLennan G, Fraser C, Ramsay CR, Vale L, Whitty P, Eccles MP, Matowe L, Shirran L, et al. Effectiveness and efficiency of guideline dissemination and implementation strategies. Health Technol Assess. 2004; 8:iii–iiv. 1–72.

26. Ahn HS, Kim HJ. Development and implementation of clinical practice guidelines: current status in Korea. J Korean Med Sci. 2012; 27:S55–S60.