Adaptive Test Model Enhancement Based on Salmon Salar Optimization and Partially Observable Markov Decision Process

Rujianto Eko Saputro, Fandy Setyo Utomo, Linda Perdana Wanti
Pages 743-757
Abstract Views 530
PDF Downloads 352
Crossref Citations -
DOI https://doi.org/10.47738/jads.v7i1.1065

Abstract


Cognitive Diagnosis Models (CDMs) in Computerized Adaptive Testing (CAT) are widely used to assess students’ cognitive abilities; however, existing approaches face significant limitations. The Latent Trait Model often suffers from specification errors due to its complexity, the Diagnostic Classification Model encounters difficulties in integrating hierarchical structures, and Deep Learning Models demand substantial computational resources. To address these challenges, this study introduces Salmon Salar Optimization (SSO) to enhance CDM performance and integrates the Partially Observable Markov Decision Process (POMDP) to improve dynamic question selection. The proposed adaptive testing framework comprises three components: preprocessing, CDM, and a selection algorithm. Experimental results on the ASSISTments 2009-2010 dataset demonstrate that SSO outperforms representative baselines from both deep learning: Neural CD and Latent Trait Model: MIRT approaches. Using 5-fold cross-validation, the proposed model achieved superior predictive performance with 75.51% accuracy and an AUC of 0.8191, highlighting its robustness compared to existing state-of-the-art methods. Furthermore, adaptive test simulations reveal that the SSO- and POMDP-based model delivers superior outcomes, attaining 80.3% accuracy with a reward of 8.03 for 10-question exams and 79.8% accuracy with a reward of 11.97 for 15-question exams. These findings confirm the effectiveness of the proposed model in enhancing cognitive diagnosis and adaptive testing performance.

Keywords


Computerized Adaptive Testing; Cognitive Diagnostic Model; Greedy Strategy; Reinforcement Learning; Selection Algorithm

Full Text:

PDF

References


H. Waheed, S. U. Hassan, N. R. Aljohani, J. Hardman, S. Alelyani, and R. Nawaz, “Predicting academic performance of students from VLE big data using deep learning models,” Comput. Human Behav., vol. 104, no. November 2019, p. 106189, 2020, doi: 10.1016/j.chb.2019.106189.

I. A. M. S. Widiastuti, I. B. N. Mantra, H. Sukoco, and M. H. Santosa, “Online assessment strategies to enhance students’ competence and their implementational challenges,” JEES (Journal English Educ. Soc., vol. 6, no. 2, pp. 245–251, 2021, doi: 10.21070/jees.v6i2.1378.

M. Adnan, A. Habib, J. Ashraf, S. Mussadiq, A. . Raza, and M. Abid, “Predicting at-Risk Students at Different Percentages of Course Length for Early Intervention Using Machine Learning Models,” IEEE Access, vol. 9, pp. 7519–7539, 2021, doi: 10.1109/ACCESS.2021.3049446.

I. Ristić, M. Runić-Ristić, T. Savić Tot, V. Tot, and M. Bajac, “The Effects and Effectiveness of An Adaptive E-Learning System on The Learning Process and Performance of Students,” Int. J. Cogn. Res. Sci. Eng. Educ., vol. 11, no. 1, pp. 77–92, 2023, doi: 10.23947/2334-8496-2023-11-1-77-92.

X. Wu, T. Xu, and R. Wu, “Research on personalized assessment Information of students’ mathematical competency based on cognitive diagnosis,” Think. Ski. Creat., vol. 53, pp. 1–13, 2024, doi: 10.1016/j.tsc.2024.101596.

Z. Shang, E. A. Erosheva, and G. Xu, “Partial-mastery cognitive diagnosis models,” Ann. Appl. Stat., vol. 15, no. 3, pp. 1529–1555, 2021, doi: 10.1214/21-AOAS1439.

J. de la Torre and J. A. Douglas, “Higher-order latent trait models for cognitive diagnosis,” Psychometrika, vol. 69, pp. 333–353, 2004, doi: 10.1007/BF02295640.

P. Zhan, W. Ma, H. Jiao, and S. Ding, “A Sequential Higher Order Latent Structural Model for Hierarchical Attributes in Cognitive Diagnostic Assessments,” Appl. Psychol. Meas., vol. 44, no. 1, pp. 65–83, 2020, doi: 10.1177/0146621619832935.

J. Guo, Z. Yan, Y. Sato, and Q. Zuo, “Salmon Salar Optimization: A Novel Natural Inspired Metaheuristic Method for Deep-Sea Probe Design for Unconventional Subsea Oil Wells,” J. Mar. Sci. Eng., vol. 12, no. 10, pp. 1–24, 2024, doi: 10.3390/jmse12101802.

Y. Hong et al., “Search-Efficient Computerized Adaptive Testing,” in Proceedings of the 32nd ACM International Conference on Information and Knowledge Managemen, Birmingham: Association for Computing Machinery, 2023, pp. 773–782. doi: 10.1145/3583780.3615049.

K. Gorney and M. D. Reckase, “Using Multiple Maximum Exposure Rates in Computerized Adaptive Testing,” J. Educ. Meas., vol. 62, no. 2, pp. 360–379, 2025, doi: 10.1111/jedm.12436.

Q. Liu et al., “Survey of Computerized Adaptive Testing: A Machine Learning Perspective,” arXiv, vol. 1, no. 1, pp. 1–35, 2024, doi: 10.48550/arXiv.2404.00712.

Y. Zhang, D. Wang, X. Gao, Y. Cai, and D. Tu, “Development of a computerized adaptive testing for internet addiction,” Front. Psychol., vol. 10, no. May, pp. 1–12, 2019, doi: 10.3389/fpsyg.2019.01010.

B. Klein and K. Kovacs, “The performance of ChatGPT and Bing on a computerized adaptive test of verbal intelligence,” PLoS One, vol. 19, no. 7 July, pp. 1–9, 2024, doi: 10.1371/journal.pone.0307097.

M. A. Samsudin, T. S. Chut, M. E. Ismail, and N. J. Ahmad, “A calibrated item bank for computerized adaptive testing in measuring science TIMSS performance,” Eurasia J. Math. Sci. Technol. Educ., vol. 16, no. 7, pp. 1–15, 2020, doi: 10.29333/EJMSTE/8259.

M. R. Harvey and C. Harrison, “Computerized adaptive testing with the I-HaND scale for monitoring patients with upper limb nerve pathology,” J. Hand Surg. (European Vol., vol. 50, no. 8, pp. 1033–1040, 2025, doi: 10.1177/17531934251324381.

H. Wang et al., “GMOCAT: A Graph-Enhanced Multi-Objective Method for Computerized Adaptive Testing,” in Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Minin, California: Association for Computing Machinery, 2023, pp. 2279–2289. doi: 10.1145/3580305.3599367.

Q. Tan, Y. Cai, F. Luo, and T. Dongbo, “Development of a High-Accuracy and Effective Online Calibration Method in CD-CAT Based on Gini Index,” J. Educ. Behav. Stat., vol. 48, no. 1, pp. 103–141, 2022, doi: 10.3102/10769986221126741.

M. She, Y. Li, D. Tu, and Y. Cai, “Computerized Adaptive Testing for Sleep Disorders: Development of an Item Bank and Validation in a Simulated Study,” Eur. J. Heal. Psychol., vol. 28, no. 3, pp. 89–100, 2021, doi: 10.1027/2512-8442/a000076.

A. D. Wulansari, D. P. Kirana, and R. Mufanti, “Development of a Computerized-Adaptive Test to Measure English Vocabulary Size with IRT,” Indones. J. Learn. Adv. Educ., vol. 5, no. 3, pp. 277–294, 2023, doi: 10.23917/ijolae.v5i3.22953.

A. S. Yaro, F. Maly, P. Prazak, and K. Maly, “Outlier Detection Performance of a Modified Z-Score Method in Time-Series RSS Observation With Hybrid Scale Estimators,” IEEE Access, vol. 12, pp. 12785–12796, 2024, doi: 10.1109/ACCESS.2024.3356731.

L. Ciccione, G. Dehaene, and S. Dehaene, “Outlier Detection and Rejection in Scatterplots: Do Outliers Influence Intuitive Statistical Judgments?,” J. Exp. Psychol. Hum. Percept. Perform., vol. 49, no. 1, pp. 129–144, 2022, doi: 10.1037/xhp0001065.

A. K. Jadiya, A. Chaudhary, and R. Thakur, “Polymorphic SBD preprocessor: A preprocessing approach for social big data,” Indian J. Comput. Sci. Eng., vol. 11, no. 6, pp. 953–961, 2020, doi: 10.21817/indjcse/2020/v11i6/201106169.

E. A. Pambudi, A. G. Fahrezi, and A. Y. Badharudin, “Forecasting Brown Sugar Production Using k-NN Minkowski Distance and Z-Score Normalization,” J. Inf. Syst. Informatics, vol. 5, no. 2, pp. 580–589, 2023, doi: 10.51519/journalisi.v5i2.485.

H. Henderi, “Comparison of Min-Max normalization and Z-Score Normalization in the K-nearest neighbor (kNN) Algorithm to Test the Accuracy of Types of Breast Cancer,” IJIIS Int. J. Informatics Inf. Syst., vol. 4, no. 1, pp. 13–20, 2021, doi: 10.47738/ijiis.v4i1.73.

S. Dai and D. Svetina Valdivia, “Dealing with Missing Responses in Cognitive Diagnostic Modeling,” Psych, vol. 4, no. 2, pp. 318–342, 2022, doi: 10.3390/psych4020028.

P. Zhan, F. Li, and H. Jiao, “Editorial: Cognitive Diagnostic Assessment for Learning,” Front. Psychol., vol. 12, no. November, pp. 10–12, 2021, doi: 10.3389/fpsyg.2021.806636.

S. Liu, H. Qian, M. Li, and A. Zhou, “QCCDM: A Q-Augmented Causal Cognitive Diagnosis Model for Student Learning,” Front. Artif. Intell. Appl., vol. 372, pp. 1536–1543, 2023, doi: 10.3233/FAIA230434.

C. Ma, J. Ouyang, and G. Xu, “Learning Latent and Hierarchical Structures in Cognitive Diagnosis Models,” Psychometrika, vol. 88, no. 1, pp. 175–207, 2023, doi: 10.1007/s11336-022-09867-5.

Y. Gu and G. Xu, “Identifiability of Hierarchical Latent Attribute Models,” Stat. Sin., vol. 33, no. 4, pp. 2561–2591, 2023, doi: 10.5705/ss.202021.0350.

X. Shi, X. Ma, W. Du, and X. Gao, “Diagnosing Chinese EFL learners’ writing ability using polytomous cognitive diagnostic models,” Lang. Test., vol. 41, no. 1, pp. 109–134, 2024, doi: 10.1177/02655322231162840.

J. Güsten, D. Berron, E. Düzel, and G. Ziegler, “Bayesian modeling of item heterogeneity in dichotomous recognition memory data and prospects for computerized adaptive testing,” Sci. Rep., vol. 12, no. 1, pp. 1–17, 2022, doi: 10.1038/s41598-022-04997-3.

Y. Choi and C. McClenen, “Development of Adaptive Formative Assessment System Using Computerized Adaptive Testing and Dynamic Bayesian Networks,” Appl. Sci., vol. 10, no. 22, p. 8196, Nov. 2020, doi: 10.3390/app10228196.

K. C. T. Han, “Components of the item selection algorithm in computerized adaptive testing,” J. Educ. Eval. Health Prof., vol. 15, no. 7, pp. 1–13, 2018, doi: 10.3352/jeehp.2018.15.7.

N. van Buuren and T. Eggen, “Latent-Class-Based Item Selection for Computerized Adaptive Progress Tests,” J. Comput. Adapt. Test., vol. 5, no. 2, pp. 22–43, 2017, doi: 10.7333/1704-0502022.

H. A. Ammar, R. Adve, S. Shahbazpanahi, G. Boudreau, and K. V. Srinivas, “POMDP-based Handoffs for User-Centric Cell-Free MIMO Networks,” in Proceedings - IEEE Global Communications Conference, GLOBECOM, Rio de Janeiro: IEEE, 2022, pp. 341–346. doi: 10.1109/GLOBECOM48099.2022.10001033.

A. Loisy and R. A. Heinonen, “Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark,” Eur. Phys. J. E, vol. 46, no. 3, pp. 1–14, 2023, doi: 10.1140/epje/s10189-023-00277-8.

M. L. Littman, “A tutorial on partially observable Markov decision processes,” J. Math. Psychol., vol. 53, no. 3, pp. 119–125, 2009, doi: 10.1016/j.jmp.2009.01.005.

Y. Wang, A. K. Bozkurt, and M. Pajic, “Reinforcement Learning with Temporal Logic Constraints for Partially-Observable Markov Decision Processes,” arXiv Cornell Univ., pp. 1–6, 2021, [Online]. Available: http://arxiv.org/abs/2104.01612

M. Zuccotto, M. Piccinelli, A. Castellini, E. Marchesini, and A. Farinelli, “Learning State-Variable Relationships in POMCP: A Framework for Mobile Robots,” Front. Robot. AI, vol. 9, no. July, pp. 1–18, 2022, doi: 10.3389/frobt.2022.819107.

J. A. Filar, Z. Qiao, and N. Ye, “POMDPs for Sustainable Fishery Management,” in 23rd International Congress on Modelling and Simulation - Supporting Evidence-Based Decision Making: The Role of Modelling and Simulation, MODSIM, Canberra: The Modelling and Simulation Society of Australia and New Zealand Inc, 2019, pp. 645–651. doi: 10.36334/modsim.2019.g2.filar.

A. A. Sulyok and K. Karacs, “Towards Using Fully Observable Policies for POMDPs,” in 2nd International Conference on Computing and Machine Intelligence (ICMI), Istanbul: IEEE, 2022, pp. 1–5. doi: 10.1109/ICMI55296.2022.9873768.

F. Wang et al., “NeuralCD: A General Framework for Cognitive Diagnosis,” IEEE Trans. Knowl. Data Eng., vol. 35, no. 8, pp. 8312–8327, 2023, doi: 10.1109/TKDE.2022.3201037.

M. D. Reckase, Multidimensional Item Response Theory. London: Springer, 2009. doi: 10.1016/S0169-7161(06)26018-8.




DOI: https://doi.org/10.47738/jads.v7i1.1065

Refbacks

  • There are currently no refbacks.


Barcode

Journal of Applied Data Sciences

ISSN:2723-6471 (Online)
Publisher:Bright Publisher
Website:http://bright-journal.org/JADS
Email:taqwa@amikompurwokerto.ac.id (principal contact)
  support@bright-journal.org (technical issues)

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0