Yıl 2018, Cilt 9, Sayı 3, Sayfalar 295 - 311 2018-09-29

Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT
Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT

Xiuzhen MAO [1] , Burhanettin OZDEMIR [2] , Yating WANG [3] , Tao XIN [4]

52 46

This study aims to investigate the effect of different item exposure controlling strategies on item selection methods in the context of multidimensional computerized adaptive testing (MCAT).  Additionally, this study aims to examine to what extend the restrictive threshold (RT) and the restrictive progressive (RPG) exposure methods suppress the item exposure rates and increase the exposure rates of underexposed items without losing psychometric precision in MCAT.  For this purpose, the performance of four item selection methods with and without exposure controls are evaluated and compared so as to determine how results differ when item exposure controlling strategies are applied with Monte-Carlo simulation method. The four item selection methods employed in this study are D-optimality, Kullback–Leibler information (KLP), the minimized error variance of linear combination score with equal weight (V1), the composite score with optimized weight (V2). On the other hand, the maximum priority index (MPI) method proposed for unidimensional CAT and two other item exposure control methods, that are RT and RPG methods proposed for cognitive diagnostic CAT, are adopted. The results show that: (1) KLP, D-optimality, and V1 performed better in recovering domain scores, and all outperformed V2 with respect to precision; (2) although V1 and V2 offer improved item bank usage rates,  KLP, D-optimality, V1, and V2 produced an unbalanced distribution of item exposure rates; (3) all exposure control strategies improve the exposure uniformity greatly and with very little loss in psychometric precision; (4) RPG and MPI perform similarly in exposure control, and outperformed RT exposure control method.

This study aims to investigate the effect of different item exposure controlling strategies on item selection methods in the context of multidimensional computerized adaptive testing (MCAT).  Additionally, this study aims to examine to what extend the restrictive threshold (RT) and the restrictive progressive (RPG) exposure methods suppress the item exposure rates and increase the exposure rates of underexposed items without losing psychometric precision in MCAT.  For this purpose, the performance of four item selection methods with and without exposure controls are evaluated and compared so as to determine how results differ when item exposure controlling strategies are applied with Monte-Carlo simulation method. The four item selection methods employed in this study are D-optimality, Kullback–Leibler information (KLP), the minimized error variance of linear combination score with equal weight (V1), the composite score with optimized weight (V2). On the other hand, the maximum priority index (MPI) method proposed for unidimensional CAT and two other item exposure control methods, that are RT and RPG methods proposed for cognitive diagnostic CAT, are adopted. The results show that: (1) KLP, D-optimality, and V1 performed better in recovering domain scores, and all outperformed V2 with respect to precision; (2) although V1 and V2 offer improved item bank usage rates,  KLP, D-optimality, V1, and V2 produced an unbalanced distribution of item exposure rates; (3) all exposure control strategies improve the exposure uniformity greatly and with very little loss in psychometric precision; (4) RPG and MPI perform similarly in exposure control, and outperformed RT exposure control method.

  • Bejar, I. I. (1991). A methodology for scoring open-ended architectural design problems. Journal of Applied Psychology, 76, 522-532.
  • Bloxom, B. M., & Vale, C. D. (1987). Multidimensional adaptive testing: a procedure for sequential estimation of the posterior centriod and dispersion of theta. Paper presented at the meeting of the Psychometric society, Montreal, Canada.
  • Bolt, D. M., & Lall, V. F. (2003). Estimation of compensatory and noncompensatory multidimensional item response models using Markov chain Monte Carlo. Applied Psychological Measurement, 27, 395-414.
  • Chang, S. W. & Twu, B. Y. (1998). A Comparative Study of Item Exposure Control Methods in Computerized Adaptive Testing. ACT Research Report Series, 98-3.
  • Chang, H. H., & Ying, Z. L. (1999). a-Stratified multistage computerized adaptive testing. Applied Psychological Measurement, 23, 211–222.
  • Chen, S. Y., Ankenmann, R. D., & Spray, J. A. (2003). The relationship between item exposure and test overlap in computerized adaptive testing. Journal of Educational Measurement, 40, 129-145.
  • Cheng, Y., & Chang, H. H. (2009). The maximum priority index method for severely constrained item selection in computerized adaptive testing. British journal of mathematical and statistical psychology, 62, 369–383.
  • Finkelman, M., Nering, M. L., & Roussos, L. A. (2009). A conditional exposure control method for multidimensional adaptive testing. Journal of Educational Measurement, 46, 84-103.
  • Huebner, A. R., Wang, C., Quinlan, K., & Seubert, L. (2015). Item exposure control for multidimensional computer adaptive testing under maximum likelihood and expected a posterior estimation. Behavior Research Methods, DOI 10.3758/s13428-015-0659-z.
  • Lee, Y. H., Ip, E. H., & Fuh, C. D. (2008). A strategy for controlling item exposure in multidimensional computerized adaptive testing. Educational and Psychological Measurement, 68, 215-232.
  • Luecht, R. M. (1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20, 389-404.
  • McKinley, R. L., & Reckase, M. D. (1982). The use of the general Rasch model with multidimensional item response data (Research Report ONR 82-1). American College Testing, Iowa City, IA. Mulder, J., & van der Linden, W. J. (2009). Multidimensional adaptive testing with optimal design criteria. Psychometrika, 74, 273-296.
  • Mulder, J. & van der Linden, W. J. (2010). Multidimensional adaptive testing with Kullback-Leibler information item selection. In W. J. van der Linden AND c. A. W. Glas (eds.), Elements of Adaptive Testing, Statistics for Social and Behaviroal Sciences, Springer Science+Businesws Media, 2010.
  • Segall, D. O. (1996). Multidimensional adaptive testing. Psychometrika, 61, 331-354.
  • Stocking, M. L. (1994). Three practical issues for modern adaptive testing item pools (ETS Research Report No. 94–5). Princeton, NJ: Educational Testing Service.
  • Stocking, M. L., & Lewis, C. (1998). Controlling item exposure conditional on ability in computerized adaptive testing. Journal of Educational and Behavioral Statistics, 23, 57–75.
  • Sympson, J. B., & Hetter, R. D. (1985). Controlling item-exposure rates in computerized adaptive testing. In Proceedings of the 27th annual meeting of the Military Testing Association (pp. 973–977). San Diego, CA: Navy Personnel Research and Development Center.
  • van der Linden, W. J. (1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics, 24, 398-412.
  • van der Linden, W. J., & Veldkamp, B. P. (2007). Conditional item exposure control in adaptive testing using item-ineligibility probabilities. Journal of Educational and Behavioral Statistics, 32, 398-418.
  • Veldkamp, B. P., & van der Linden, W. J. (2002). Multidimensional adaptive testing with constraints on test content. Psychometrika, 67, 575-588.
  • Wang, C., & Chang, H. H. (2011). Item selection in multidimensional computerized adaptive testing-gaining information from different angles. Psychometrika, 76, 363-384.
  • Wang, C., Chang, H. H., & Boughton, K. A. (2011). Kullback-Leibler information and its applications in multidimensional adaptive testing. Psychometrika, 76, 13-39.
  • Wang, C., Chang, H. H., & Boughton, K. A. (2013). Deriving stopping rules for multidimensional computerized adaptive testing. Applied Psychological Measurement, 37(2), 99-122.
  • Wang, C., Chang, H. H., & Huebner, A. (2011). Restrictive stochastic item selection methods in cognitive diagnostic computerized adaptive testing. Journal of Educational Measurement, 48, 255-273.
  • Wang, W. C., & Chen, P. H. (2004). Implementation and measurement efficiency of multidimensional computerized adaptive testing. Applied Psychologica Measurement, 28, 295–316.
  • Yao, L. (2010). Reporting valid and reliability overall score and domain scores. Journal of Educational Measurement, 47, 339-360.
  • Yao, L. (2012). Multidimensional CAT item selection methods for domain scores and composite scores: theory and applications. Psychometrika, 77, 495-523.
  • Yao, L. (2014a). Multidimensional CAT item selection methods for domain scores and composite scores with item exposure control and content constrains. Journal of Educational Measurement, 51,18-38.
  • Yao, L. (2014b). Multidimensional item response theory for score reporting. In Cheng, Y., & Chang, H.-H. (Eds.), Advances in modern international testing:Transition from summative to formative assessment. Charlotte, NC: Information Age.
  • Yao, L., Pommerich, M., & Segall, D. O. (2014). Using Multidimensional CAT to Administer a Short, Yet Precise, Screening Test. Applied Psychological Measurement, 38,614-631.
  • Yao, L., & Schwarz, R. D. (2006). A multidimensional partial credit model with associated item and test statistics: An application to mixed-format tests. Applied Psychological Measurement, 37, 3-23.
Birincil Dil en
Konular Sosyal
Dergi Bölümü Makaleler
Yazarlar

Orcid: 0000-0001-8245-3633
Yazar: Xiuzhen MAO
Ülke: China


Orcid: 0000-0001-7716-2700
Yazar: Burhanettin OZDEMIR
Kurum: Siirt University, TURKEY
Ülke: Turkey


Orcid: 0000-0001-9328-5380
Yazar: Yating WANG
Ülke: China


Orcid: 0000-0002-2308-8557
Yazar: Tao XIN
Ülke: China


Bibtex @araştırma makalesi { epod407521, journal = {Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi}, issn = {1309-6575}, eissn = {1309-6575}, address = {Eğitimde ve Psikolojide Ölçme ve Değerlendirme Derneği}, year = {2018}, volume = {9}, pages = {295 - 311}, doi = {10.21031/epod.407521}, title = {Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT}, key = {cite}, author = {WANG, Yating and MAO, Xiuzhen and XIN, Tao and OZDEMIR, Burhanettin} }
APA MAO, X , OZDEMIR, B , WANG, Y , XIN, T . (2018). Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT. Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi, 9 (3), 295-311. DOI: 10.21031/epod.407521
MLA MAO, X , OZDEMIR, B , WANG, Y , XIN, T . "Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT". Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi 9 (2018): 295-311 <http://dergipark.gov.tr/epod/issue/39489/407521>
Chicago MAO, X , OZDEMIR, B , WANG, Y , XIN, T . "Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT". Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi 9 (2018): 295-311
RIS TY - JOUR T1 - Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT AU - Xiuzhen MAO , Burhanettin OZDEMIR , Yating WANG , Tao XIN Y1 - 2018 PY - 2018 N1 - doi: 10.21031/epod.407521 DO - 10.21031/epod.407521 T2 - Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi JF - Journal JO - JOR SP - 295 EP - 311 VL - 9 IS - 3 SN - 1309-6575-1309-6575 M3 - doi: 10.21031/epod.407521 UR - http://dx.doi.org/10.21031/epod.407521 Y2 - 2018 ER -
EndNote %0 Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT %A Xiuzhen MAO , Burhanettin OZDEMIR , Yating WANG , Tao XIN %T Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT %D 2018 %J Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi %P 1309-6575-1309-6575 %V 9 %N 3 %R doi: 10.21031/epod.407521 %U 10.21031/epod.407521
ISNAD MAO, Xiuzhen , OZDEMIR, Burhanettin , WANG, Yating , XIN, Tao . "Investigating The Effect of Exposure-Control Strategies on Item Selection Methods in MCAT". Eğitimde ve Psikolojide Ölçme ve Değerlendirme Dergisi 9 / 3 (Eylül 2018): 295-311. http://dx.doi.org/10.21031/epod.407521