J Educ Eval Health Prof.  2017;14:19. 10.3352/jeehp.2017.14.19.

Sequential Objective Structured Clinical Examination based on item response theory in Iran

• Hejri SM
• Jalili M

• KMID: 2406682
• DOI: http://doi.org/10.3352/jeehp.2017.14.19

Abstract

PURPOSE
In a sequential objective structured clinical examination (OSCE), all students initially take a short screening OSCE. Examinees who pass are excused from further testing, but an additional OSCE is administered to the remaining examinees. Previous investigations of sequential OSCE were based on classical test theory. We aimed to design and evaluate screening OSCEs based on item response theory (IRT).
METHODS
We carried out a retrospective observational study. At each station of a 10-station OSCE, the students' performance was graded on a Likert-type scale. Since the data were polytomous, the difficulty parameters, discrimination parameters, and students' ability were calculated using a graded response model. To design several screening OSCEs, we identified the 5 most difficult stations and the 5 most discriminative ones. For each test, 5, 4, or 3 stations were selected. Normal and stringent cut-scores were defined for each test. We compared the results of each of the 12 screening OSCEs to the main OSCE and calculated the positive and negative predictive values (PPV and NPV), as well as the exam cost.
RESULTS
A total of 253 students (95.1%) passed the main OSCE, while 72.6% to 94.4% of examinees passed the screening tests. The PPV values ranged from 0.98 to 1.00, and the NPV values ranged from 0.18 to 0.59. Two tests effectively predicted the results of the main exam, resulting in financial savings of 34% to 40%.
CONCLUSION
If stations with the highest IRT-based discrimination values and stringent cut-scores are utilized in the screening test, sequential OSCE can be an efficient and convenient way to conduct an OSCE.

Keyword

Sequential; Objective structured clinical examination; Item response theory; Cost

MeSH Terms

Discrimination (Psychology)
Humans
Income
Iran*
Mass Screening
Observational Study
Retrospective Studies

Figure

• 

  Fig. 1. The item characteristic curves for each of the 10 stations in the main objective structured clinical examination.

• 

  Fig. 2. The test information curve (upper curve) and standard error curve (lower curve) of the main objective structured clinical examination.

Reference

References

1. Harden RM. Misconceptions and the OSCE. Med Teach. 2015; 37:608–610. https://doi.org/10.3109/0142159X.2015.1042443.

2. Gormley G. Summative OSCEs in undergraduate medical education. Ulster Med J. 2011; 80:127–132.

3. Pell G, Fuller R, Homer M, Roberts T. Advancing the objective structured clinical examination: sequential testing in theory and practice. Med Educ. 2013; 47:569–577. https://doi.org/10.1111/medu.12136.

4. Currie GP, Sivasubramaniam S, Cleland J. Sequential testing in a high stakes OSCE: determining number of screening tests. Med Teach. 2016; 38:708–714. https://doi.org/10.3109/0142159X.2015.1079309.

5. Mortaz Hejri S, Yazdani K, Labaf A, Norcini JJ, Jalili M. Introducing a model for optimal design of sequential objective structured clinical examinations. Adv Health Sci Educ Theory Pract. 2016; 21:1047–1060. https://doi.org/10.1007/s10459-016-9673-x.

6. De Champlain AF. A primer on classical test theory and item response theory for assessments in medical education. Med Educ. 2010; 44:109–117. https://doi.org/10.1111/j.1365-2923.2009.03425.x.

7. Colliver JA, Mast TA, Vu NV, Barrows HS. Sequential testing with a performance-based examination using standardized patients. Acad Med. 1991; 66(9 Suppl):S64–S66. https://doi.org/10.1097/00001888-199109001-00023.

8. Colliver JA, Vu NV, Barrows HS. Screening test length for sequential testing with a standardized-patient examination: a receiver operating characteristic (ROC) analysis. Acad Med. 1992; 67:592–595. https://doi.org/10.1097/00001888-199209000-00010.

9. Rothman AI, Blackmore DE, Dauphinee WD, Reznick R. Tests of sequential testing in two years’ results of part 2 of the Medical Council of Canada Qualifying Examination. Acad Med. 1997; 72(10 Suppl 1):S22–S24. https://doi.org/10.1097/00001888-199710001-00008.

10. Smee SM, Dauphinee WD, Blackmore DE, Rothman AI, Reznick RK, Des Marchais J. A sequenced OSCE for licensure: administrative issues, results and myths. Adv Health Sci Educ Theory Pract. 2003; 8:223–236.

11. Cookson J, Crossley J, Fagan G, McKendree J, Mohsen A. A final clinical examination using a sequential design to improve cost-effectiveness. Med Educ. 2011; 45:741–747. https://doi.org/10.1111/j.1365-2923.2010.03926.x.