Psychometric Validation of an AI-Based Evaluation System for Identifying Discrepancies in Learning Processes

P. Wayan Arta Suyasa, I Gusti Ngurah Pujawan, Dewa Gede Hendra Divayana, I Dewa Ayu Made Budhyani, I Made Sugiarta, I Made Candiasa
Pages 1149-1162
Abstract Views 585
PDF Downloads 415
Crossref Citations -
DOI https://doi.org/10.47738/jads.v7i2.1168

Abstract


This research advances the field of educational evaluation by designing and psychometrically validating an artificial intelligence (AI)- based diagnostic tool to detect discrepancies in university learning processes. The main novelty is the integration of the Provus Discrepancy Model combined with a forward-chaining inference engine. This research aims to transform evaluation from an administrative activity to an ongoing process of improvement. The tool was developed and validated through a sequential mixed-methods approach with 400 participants from 3 state universities and 8 evaluation experts. Results from the study provide evidence that the validated system created a substantial range of psychometric characteristics. These psychometric characteristics include strong content validity (SD-CVI/Ave = 0.94); high internal consistency and reliability (Cronbach's α = 0.94); solid construct validity as demonstrated through Confirmatory Factor Analysis (CFA) (CFI = 0.94; RMSEA = 0.054) and a substantial range of predictive analytics (diagnostic learning analytics), which the AI learning analytics engine evaluated learning discrepancies with a 92.4% diagnostic accuracy (47.4% more accurate than manual evaluation methods). The system's validated usefulness is demonstrated through high system usability (SUS = 88.2); high practical utility (85% total score on the Pragmatic Utility Assessment); significant utility (real-world) practical utility (detected 45 discrepancy patterns), cost efficiency (73% cost and 67% analysis time compared to traditional methods), and a range of analytics (predictive and learning discrepancy analytics). The significant contribution of this study is the development of the world's first integrated AI evaluation system that meets high methodological and psychometric standards, along with a set of real-time diagnostic analytics. Ultimately, this study developed the first truly integrated, novel paradigm evaluation system that combined the historically established evaluation construct and mechanisms with the most advanced AI capabilities, providing educators and institutions with evaluation tools to deliver data-driven pedagogical strategies and interventions in higher education. 


Keywords


Design; E-Evaluation Instrument; Psychometric Validity; Artificial Intelligence; Learning Discrepancy.

Full Text:

PDF

References


R. Arroyo González, E. Fernández-Lancho, and J. de la Hoz-Ruiz, “Technologies for Learning Writing in L1 and L2 for the 21st Century: Effects on Writing Metacognition, Self-Efficacy and Argumentative Structuring,” Journal of Information Technology Education: Research, vol. 20, pp. 087–116, 2021, doi: 10.28945/4705.

X. Zhang and H. Wu, “Investigating Structural Model Fit Evaluation,” Struct Equ Modeling, vol. 31, no. 5, pp. 863–881, Sep. 2024, doi: 10.1080/10705511.2024.2350023.

D. G. H. Divayana, P. W. A. Suyasa, and N. K. Widiartini, “An innovative model as evaluation model for information technology-based learning at ICT vocational schools,” Heliyon, vol. 7, no. 2, p. e06347, Feb. 2021, doi: 10.1016/j.heliyon.2021.e06347.

I. P. W. Ariawan, W. Sugandini, I. M. Ardana, G. A. D. Sugiharni, A. W. O. Gama, and D. G. H. Divayana, “Forms and Field Trials of a Digital Evaluation Tool: Integrating F-S Model, WP Method, and Balinese Local Wisdom for Effective E-Learning,” Journal of Applied Data Sciences, vol. 5, no. 2, pp. 441–454, May 2024, doi: 10.47738/jads.v5i2.201.

T. Wulandari, A. Widiastuti, N. Nasiwan, J. Setiawan, M. R. Fadli, and H. Hadisaputra, “Development of learning models for inculcating Pancasila values,” International Journal of Evaluation and Research in Education (IJERE), vol. 12, no. 3, p. 1364, Sep. 2023, doi: 10.11591/ijere.v12i3.25687.

D. G. H. Divayana et al., “Utilization of the Weighted Product-Based CIPP Evaluation Model in Determining the Best Online Platform,” HighTech and Innovation Journal, vol. 4, no. 1, pp. 233–248, Mar. 2023, doi: 10.28991/HIJ-2023-04-01-015.

D. G. H. Divayana, P. W. A. Suyasa, I. P. W. Ariawan, N. W. R. Mariani, G. A. D. Sugiharni, and A. W. O. Gama, “User Interface Design for DIVAYANA Evaluation Application Based on Positive-Negative Discrepancy,” Journal of Applied Data Sciences, vol. 4, no. 4, pp. 317–332, Dec. 2023, doi: 10.47738/jads.v4i4.136.

P. W. A. Suyasa, D. G. H. Divayana, I. P. W. Ariawan, M. S. L. Andayani, I. N. I. Wiradika, and A. Adiarta, “Field trial of Provus-Alkin-amalgamation evaluation application based on Weighted-Product-Rwa-Bhineda mods,” Journal of Education and Learning (EduLearn), vol. 19, no. 1, pp. 495–505, Feb. 2025, doi: 10.11591/edulearn.v19i1.21114.

P. W. A. Suyasa, D. G. H. Divayana, and I. P. W. Ariawan, “User interface design of Amalgamation evaluation application based on Provus-Alkin-WP integrated with Rwa Bhineda,” in The 8th International Conference on Technology And Vocational Teachers 2022, Yogyakarta: AIP Publishing, Jul. 2024. doi: 10.1063/5.0214097.

M. L. Magruder, M. Miskiewicz, A. N. Rodriguez, M. Ng, and A. Abdelgawad, “Comparison of ChatGPT plus (version 4.0) and pretrained AI model (Orthopod) on orthopaedic in-training exam (OITE),” The Surgeon, vol. 23, no. 3, pp. 187–191, Jun. 2025, doi: 10.1016/j.surge.2025.04.004.

V. Santos, J. Teles, and P. Quaresma, “Exploring Quadrilaterals: An Interactive Task for 7th Grade Students Using GeoGebra Classroom,” International Journal for Technology in Mathematics Education, vol. 31, no. 3, pp. 107–116, Sep. 2024, doi: 10.1564/tme_v31.3.01.

M. Abdelshiheed, T. Barnes, and M. Chi, “How and When: The Impact of Metacognitive Knowledge Instruction and Motivation on Transfer Across Intelligent Tutoring Systems,” Int J Artif Intell Educ, vol. 34, no. 3, pp. 974–1007, Sep. 2024, doi: 10.1007/s40593-023-00371-0.

R. Branscum, C. J. Eck, K. N. Marsh, and B. M. Coleman, “Instructional Practice Needs of Oklahoma Agricultural Educators by Career Phase,” J Agric Educ, vol. 66, no. 3, p. 8, Jul. 2025, doi: 10.5032/jae.v66i3.3024.

A. E. Hallaran and C. K. Voulgarides, “Countering the Construction of Learning Disability: A Qualitative Investigation at the Meso-Level of Policy,” Learning Disabilities: A Multidisciplinary Journal, Dec. 2024, doi: 10.18666/LDMJ-2024-V29-I2-12647.

J. A. Hogan, “Discrepancy model to RtI : Gauging teacher preparedness for this shift in specific learning disability classification,” Journal of Research in Special Educational Needs, vol. 25, no. 2, pp. 379–387, Apr. 2025, doi: 10.1111/1471-3802.12730.

W. Walker-Schmidt, C. Kaul, and L. Crocker Papadakis, “Onboarding Effects on Engagement and Retention in the IT Sector,” Impacting Education: Journal on Transforming Professional Practice, vol. 7, no. 4, pp. 8–15, Nov. 2022, doi: 10.5195/ie.2022.220.

X. Liu and Y. Xiao, “Chinese university teachers’ engagement with generative AI in different stages of foreign language teaching: A qualitative enquiry through the prism of ADDIE,” Educ Inf Technol (Dordr), vol. 30, no. 1, pp. 485–508, Jan. 2025, doi: 10.1007/s10639-024-13117-9.

D. Huang, Y. Huang, and J. J. Cummings, “Exploring the integration and utilisation of generative AI in formative e-assessments: A case study in higher education,” Australasian Journal of Educational Technology, vol. 40, no. 4, pp. 1–19, Sep. 2024, doi: 10.14742/ajet.9467.

M. Ma, D. T. K. Ng, Z. Liu, and G. K. W. Wong, “Fostering responsible AI literacy: A systematic review of K-12 AI ethics education,” Computers and Education: Artificial Intelligence, vol. 8, p. 100422, Jun. 2025, doi: 10.1016/j.caeai.2025.100422.

E. C. Villaceran and C. M. Himang, “Data on behavioural intention to use AI copilot through TAM and AI ecological education policy lens,” Data Brief, vol. 61, p. 111686, Aug. 2025, doi: 10.1016/j.dib.2025.111686.

A. Bin Nofal et al., “AI-enhanced interview simulation in the metaverse: Transforming professional skills training through VR and generative conversational AI,” Computers and Education: Artificial Intelligence, vol. 8, p. 100347, Jun. 2025, doi: 10.1016/j.caeai.2024.100347.

E. Hogan and Y. Sun, “The Association Between Classroom Dialogic Interaction and Student Reading Performance: A Mixed Methods Study of Teacher Stance, Discourse Moves, and Reading Achievement,” Read Res Q, vol. 60, no. 2, Apr. 2025, doi: 10.1002/rrq.70009.

L. P. Bailes, S. Ahmad, M. Saylor, and M. N. Vitale, “Quality or Control: High-Needs Principals’ Perceptions of a PSEL-Based Evaluation System,” Journal of Research on Leadership Education, vol. 18, no. 4, pp. 622–648, Dec. 2023, doi: 10.1177/19427751221118952.

D. Dukpa, S. Carrington, and S. Mavropoulou, “Bhutanese Teachers’ Views about the Inclusion of Students on the Autism Spectrum,” Intl J Disabil Dev Educ, vol. 71, no. 2, pp. 251–269, Feb. 2024, doi: 10.1080/1034912X.2022.2095357.

A. Setiawan, “An exploratory sequential mixed-methods approach to understanding students’ entrepreneurial self-efficacy,” Journal of Turkish Science Education, vol. 20, no. 2, pp. 320–332, Jul. 2023, doi: 10.36681/tused.2023.018.

Y. Wang and M. Kruk, “Modeling the interaction between teacher credibility, teacher confirmation, and English major students’ academic engagement: A sequential mixed-methods approach,” Studies in Second Language Learning and Teaching, vol. 14, no. 2, pp. 235–265, Jan. 2024, doi: 10.14746/ssllt.38418.

D. G. H. Divayana, A. Adiarta, and P. W. A. Suyasa, “Development of Discrepancy Evaluation Model based on Tat Twam Asi with TOPSIS Calculation,” International Journal of Advanced Computer Science and Applications, vol. 13, no. 7, 2022, doi: 10.14569/IJACSA.2022.0130787.

P. W. A. Suyasa and D. G. H. Divayana, “Instrumen Evaluasi Model Discrepancy-CSE-UCLA Dalam Rangka Menunjang Evaluasi Efektivitas Pelaksanaan Pembelajaran Synchronous,” Jurnal Pedagogi dan Pembelajaran, vol. 5, no. 2, pp. 197–207, Jul. 2022, doi: 10.23887/jp2.v5i2.48447.

I. P. W. Ariawan, P. W. A. Suyasa, A. Adiarta, I. K. G. Sukawijana, N. Santiyadnya, and D. G. H. Divayana, “User Interface Design of Digital Test Based on Backward Chaining as a Measuring Tool for Students’ Critical Thinking,” International Journal of Advanced Computer Science and Applications, vol. 16, no. 1, 2025, doi: 10.14569/IJACSA.2025.0160156.

D. G. H. Divayana, P. W. A. Suyasa, and I. P. W. Ariawan, “Amalgamation evaluation model design based on modification weighted product-Provus-Alkin-Rwa Bhineda,” International Journal of Evaluation and Research in Education (IJERE), vol. 13, no. 4, p. 2068, Aug. 2024, doi: 10.11591/ijere.v13i4.27712.

D. G. H. Divayana, A. Adiarta, and I. G. Sudirtha, “Instruments Development of Tri Kaya Parisudha-Based Countenance Model in Evaluating the Blended Learning,” International Journal of Engineering Pedagogy (iJEP), vol. 9, no. 5, p. 55, Nov. 2019, doi: 10.3991/ijep.v9i5.11055.

G. A. D. Sugiharni and N. W. Setiasih, “Validitas dan Reliabilitas Instrumen Evaluasi Blended Learning Matakuliah Matematika Diskrit di STIKOM Bali Berbasis Model Alkin,” IndoMath: Indonesia Mathematics Education, vol. 1, no. 2, p. 93, Aug. 2018, doi: 10.30738/indomath.v1i2.2626.

P. W. A. Suyasa, D. G. H. Divayana, I. P. W. Ariawan, M. S. L. Andayani, I. N. I. Wiradika, and A. Adiarta, “Field trial of Provus-Alkin-amalgamation evaluation application based on Weighted-Product-Rwa-Bhineda mods,” Journal of Education and Learning (EduLearn), vol. 19, no. 1, pp. 495–505, Feb. 2025, doi: 10.11591/edulearn.v19i1.21114.




DOI: https://doi.org/10.47738/jads.v7i2.1168

Refbacks

  • There are currently no refbacks.


Barcode

Journal of Applied Data Sciences

ISSN:2723-6471 (Online)
Publisher:Bright Publisher
Website:http://bright-journal.org/JADS
Email:taqwa@amikompurwokerto.ac.id (principal contact)
  support@bright-journal.org (technical issues)

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0