J Educ Eval Health Prof.  2018;15:14. 10.3352/jeehp.2018.15.14.

Post-hoc simulation study of computerized adaptive testing for the Korean Medical Licensing Examination

• Seo DG
• Choi J

• KMID: 2439270
• DOI: http://doi.org/10.3352/jeehp.2018.15.14

Abstract

PURPOSE
Computerized adaptive testing (CAT) has been adopted in licensing examinations because it improves the efficiency and accuracy of the tests, as shown in many studies. This simulation study investigated CAT scoring and item selection methods for the Korean Medical Licensing Examination (KMLE).
METHODS
This study used a post-hoc (real data) simulation design. The item bank used in this study included all items from the January 2017 KMLE. All CAT algorithms for this study were implemented using the "˜catR' package in the R program.
RESULTS
In terms of accuracy, the Rasch and 2-parametric logistic (PL) models performed better than the 3PL model. The "˜modal a posteriori' and "˜expected a posterior' methods provided more accurate estimates than maximum likelihood estimation or weighted likelihood estimation. Furthermore, maximum posterior weighted information and minimum expected posterior variance performed better than other item selection methods. In terms of efficiency, the Rasch model is recommended to reduce test length.
CONCLUSION
Before implementing live CAT, a simulation study should be performed under varied test conditions. Based on a simulation study, and based on the results, specific scoring and item selection methods should be predetermined.

Keyword

Algorithms; Computers; Korea; Logistic models; Research design

MeSH Terms

Animals
Cats
Korea
Licensure*
Logistic Models
Research Design

Reference

References

1. Kim CH. Presidential address: preparing for permanent test centers and computerized adaptive testing. J Educ Eval Health Prof. 2018; 15:1. https://doi.org/10.3352/jeehp.2018.15.1.

2. Wang T, Vispoel WP. Properties of ability estimation methods in computerized adaptive testing. J Educ Meas. 1998; 35:109–135. https://doi.org/10.1111/j.1745-3984.1998.tb00530.x.

3. Weiss DJ, McBride JR. Bias and information of Bayesian adaptive testing. Appl Psychol Meas. 1984; 8:273–285. https://doi.org/10.1177/014662168400800303.

4. Weiss DJ. Better data from better measurements using computerized adaptive testing. J Methods Meas Soc Sci. 2011; 2:1–27. https://doi.org/10.2458/v2i1.12351.

5. Veerkamp WJ, Berger MP. Some new item selection criteria for adaptive testing. J Educ Behav Stat. 1997; 22:203–226. https://doi.org/10.2307/1165378.

6. Van der Linden WJ. Bayesian item selection criteria for adaptive testing. Psychometrika. 1998; 63:201–216. https://doi.org/10.1007/bf02294775.

7. Magis D, Raiche G. Random generation of response patterns under computerized adaptive testing with the R package catR. J Stat Softw. 2012; 48:1–31. https://doi.org/10.18637/jss.v048.i08.

8. R Development Core Team. R: a language and environment for statistical computing [Internet]. Vienna: R Foundation for Statistical Computing. 2008. [cited 2018 Apr 13]. Available from: http://www.R-project.org.

9. Kingsbury GG, Zara AR. Procedures for selecting items for computerized adaptive tests. Appl Meas Educ. 1989; 2:359–375. https://doi.org/10.1207/s15324818ame0204_6.

10. Warm TA. Weighted likelihood estimation of ability in item response theory. Psychometrika. 1989; 54:427–450. https://doi.org/10.1007/bf02294627.

11. Bock RD, Mislevy RJ. Adaptive EAP estimation of ability in a microcomputer environment. Appl Psychol Meas. 1982; 6:431–444. https://doi.org/10.1177/014662168200600405.

12. Lord FM. A broad-range tailored test of verbal ability. Appl Psychol Meas. 1977; 1:95–100. https://doi.org/10.1177/014662167700100115.

13. Owen RJ. A Bayesian sequential procedure for quantal response in the context of adaptive mental testing. J Am Stat Assoc. 1975; 70:351–356. https://doi.org/10.2307/2285821.

14. Chang HH, Ying Z. A global information approach to computerized adaptive testing. Appl Psychol Meas. 1996; 20:213–229. https://doi.org/10.1177/014662169602000303.

15. Lim M. A validation study about standard setting for 2014 medical licensing examination. Seoul: Korea Health Personnel Licensing Examination Institute;2014.

16. Zhang J. Conditional covariance theory and detect for polytomous items. Psychometrika. 2007; 72:69–91. https://doi.org/10.1007/s11336-004-1257-7.

17. Folk VG, Green BF. Adaptive estimation when the unidimensionality assumption of IRT is violated. Appl Psychol Meas. 1989; 13:373–390. https://doi.org/10.1177/014662168901300404.