J Educ Eval Health Prof.  2018;15:23. 10.3352/jeehp.2018.15.23.

Agreement between 2 raters' evaluations of a traditional prosthodontic practical exam integrated with directly observed procedural skills in Egypt

• Khalifa AK
• Hegazy S

• KMID: 2439279
• DOI: http://doi.org/10.3352/jeehp.2018.15.23

Abstract

PURPOSE
This study aimed to assess the agreement between 2 raters in evaluations of students on a prosthodontic clinical practical exam integrated with directly observed procedural skills (DOPS).
METHODS
A sample of 76 students was monitored by 2 raters to evaluate the process and the final registered maxillomandibular relation for a completely edentulous patient at Mansoura Dental School, Egypt on a practical exam of bachelor's students from May 15 to June 28, 2017. Each registered relation was evaluated from a total of 60 marks subdivided into 3 score categories: occlusal plane orientation (OPO), vertical dimension registration (VDR), and centric relation registration (CRR). The marks for each category included an assessment of DOPS. The marks of OPO and VDR for both raters were compared using the graph method to measure reliability through Bland and Altman analysis. The reliability of the CRR marks was evaluated by the Krippendorff alpha ratio.
RESULTS
The results revealed highly similar marks between raters for OPO (mean= 18.1 for both raters), with close limits of agreement (0.73 and −0.78). For VDR, the mean marks were close (mean= 17.4 and 17.1 for examiners 1 and 2, respectively), with close limits of agreement (2.7 and −2.2). There was a strong correlation (Krippendorff alpha ratio, 0.92; 95% confidence interval, 0.79-0.99) between the raters in the evaluation of CRR.
CONCLUSION
The 2 raters' evaluation of a clinical traditional practical exam integrated with DOPS showed no significant differences in the evaluations of candidates at the end of a clinical prosthodontic course. The limits of agreement between raters could be optimized by excluding subjective evaluation parameters and complicated cases from the examination procedure.

Keyword

Educational assessment; Dental school; Prosthodontics; Educational measurement; Dental education

MeSH Terms

Centric Relation
Dental Occlusion
Education, Dental
Educational Measurement
Egypt*
Humans
Methods
Prosthodontics
Schools, Dental
Vertical Dimension

Figure

• 

  Fig. 1. Pre-prepared evaluation sheet.

• 

  Fig. 2. Plots of differences between OPO marks for both raters versus the means of the marks. OPO, occlusal plane orientation; SD, standard deviation.

• 

  Fig. 3. Plots of differences between VDR marks for both raters versus the means of the marks. VDR, vertical dimension registration; SD, standard deviation.

Reference

References

1. Evgeniou E, Peter L, Tsironi M, Iyer S. Assessment methods in surgical training in the United Kingdom. J Educ Eval Health Prof. 2013; 10:2. https://doi.org/10.3352/jeehp.2013.10.2.

2. Bosse HM, Mohr J, Buss B, Krautter M, Weyrich P, Herzog W, Junger J, Nikendei C. The benefit of repetitive skills training and frequency of expert feedback in the early acquisition of procedural skills. BMC Med Educ. 2015; 15:22. https://doi.org/10.1186/s12909-015-0286-5.

3. Zayyan M. Objective structured clinical examination: the assessment of choice. Oman Med J. 2011; 26:219–222. https://doi.org/10.5001/omj.2011.55.

4. Tricio J, Woolford M, Thomas M, Lewis-Greene H, Georghiou L, Andiappan M, Escudier M. Dental students’ peer assessment: a prospective pilot study. Eur J Dent Educ. 2015; 19:140–148. https://doi.org/10.1111/eje.12114.

5. Al-Dwairi ZN. Complete edentulism and socioeconomic factors in a Jordanian population. Int J Prosthodont. 2010; 23:541–543.

6. McMillan W. Making the most of teaching at the chairside. Eur J Dent Educ. 2011; 15:63–68. https://doi.org/10.1111/j.1600-0579.2010.00638.x.

7. Giavarina D. Understanding Bland Altman analysis. Biochem Med (Zagreb). 2015; 25:141–151. https://doi.org/10.11613/BM.2015.015.

8. Yong AG, Pearce S. A beginner’s guide to factor analysis: focusing on exploratory factor analysis. Tutor Quant Methods Psychol. 2013; 9:79–94. https://doi.org/10.20982/tqmp.09.2.p079.

9. Bidra AS. Three-dimensional esthetic analysis in treatment planning for implant-supported fixed prosthesis in the edentulous maxilla: review of the esthetics literature. J Esthet Restor Dent. 2011; 23:219–236. https://doi.org/10.1111/j.1708-8240.2011.00428.x.

10. Shetty S, Shenoy KK, Ninan J, Mahaseth P. An evaluation of relation between the relative parallelism of occlusal plane to ala-tragal line and variation in the angulation of Po-Na-ANS angle in dentulous subjects: A cephalometric study. J Indian Prosthodont Soc. 2015; 15:168–172. https://doi.org/10.4103/0972-4052.159968.

11. Custodio W, Gomes SG, Faot F, Garcia RC, Del Bel Cury AA. Occlusal force, electromyographic activity of masticatory muscles and mandibular flexure of subjects with different facial types. J Appl Oral Sci. 2011; 19:343–349. https://doi.org/10.1590/s1678-77572011005000008.

12. Abduo J, Lyons K. Clinical considerations for increasing occlusal vertical dimension: a review. Aust Dent J. 2012; 57:2–10. https://doi.org/10.1111/j.1834-7819.2011.01640.x.

13. Ahlers MO, Jakstat H. Development of a computer-assisted system for model-based condylar position analysis (E-CPM). Int J Comput Dent. 2009; 12:223–234.

14. Al-Wardy NM. Assessment methods in undergraduate medical education. Sultan Qaboos Univ Med J. 2010; 10:203–209.

15. Tichon JG, Wallis GM. Stress training and simulator complexity: why sometimes more is less. Behav Inf Technol. 2010; 29:459–466. https://doi.org/10.1080/01449290903420184.