The evaluation of students' attainment of course learning outcomes is a fundamental aspect of a successful engineering program, exemplified by CDIO Standard 11 'Learning Assessment'. However, earlier research has identified a prevailing gap in assessment competency among faculty. Rooted in established interdisciplinary concepts and theories, this study aims to explore the usage of ChatGPT-4 as a co-pilot to guide faculty in assessment design refinement. To achieve this goal, we adopt a conversational analysis approach, contextualizing our study within the settings of the final exam of the senior course “Wireless Sensor Networks”, offered at ESPRIT School of Engineering. We propose a framework to guide the implementation of the conversational analysis method. Our research results illustrate the merits, potentials, and limitations of using ChatGPT as a co-pilot to assist faculty in refining the assessment design process. It also brings into evidence the importance of keeping a ‘human in the loop’ perspective during the faculty-ChatGPT assessment co-creation activities. Our study can pave the way for further research on other potential applications of “Human-AI co-creation” and augmented man-machine intelligence in a CDIO engineering education.
Affiliations:
Esprit School of Business, Tunisia
ESPRIT School of Engineering, Tunisia
Keywords:
Learning Assessment
CDIO
Generative AI
ChatGPT
Engineering education
Co-Creation
CDIO Standard 10
CDIO Standard 11
Year:
2024
Reference:
Abedin, B., Meske, C., Junglas, I., Rabhi, F. & Motahari-Nezhad, H.R. (2022). Designing and Managing Human-AI interactions. Information Systems Frontiers, 24, (pp.691–697). https://doi.org/10.1007/s10796-022-10313-1 :
https://doi.org/10.1007/s10796-022-10313-1
Algabri, H. K., Kharade, K. G., & Kamat, R. K. (2021). Promise, Threats, And Personalization In Higher Education with Artificial Intelligence. Webology, 18(6), (pp. 2129–2139).:
Ali, Q. I. (2023). Surveying Different Student Outcome Assessment Methods for ABET Accredited Computer Engineering Programs. Research Reports on Computer Science, 2(1), (pp. 56–76). :
https://doi.org/10.37256/rrcs.2120232577
Bond, M., Khosravi, H., De Laat, M., Bergdahl, N., Negrea, V., Oxley, E., Pham, P., Chong, S.W., & Siemens, G. (2023). A Meta Systematic Review of Artificial Intelligence in Higher Education: A Call for Increased Ethics, Collaboration, and Rigour. Pre-print.:
https://doi.org/10.1186/s41239-023-00436-z
CDIO (2020). CDIO Standards 3.0, 10 June 2020. Available at http://cdio.org/files/CDIO%20STANDARDS%203.pdf. :
CDIO. (2022). CDIO Optional Standards 3.0. 12 July 2022. Available at http://www.cdio.org/content/cdio-optional-standards-30 :
Chaudhry, I. S., Sarwary, S. A. M., El Refae, G. A., & Chabchoub, H. (2023). Time to Revisit Existing Student’s Performance Evaluation Approach in Higher Education Sector in a New Era of ChatGPT—A Case Study. Cogent Education, 10(1), 2210461. https://doi.org/10.1080/2331186X.2023.2210461 :
https://doi.org/10.1080/2331186X.2023.2210461
Cheung, B. H. H., Lau, G. K. K., Wong, G. T. C., Lee, E. Y. P., Kulkarni, D., Seow, C. S., ... & Co, M. T. H. (2023). ChatGPT Versus Human in Generating Medical Graduate Exam Multiple Choice Questions—A Multinational Prospective Ptudy (Hong Kong SAR, Singapore, Ireland, and the United Kingdom). PLoS One, 18(8), e0290691. https://doi.org/10.1371/journal.pone.0290691 :
https://doi.org/10.1371/journal.pone.0290691
Daun, M., & Brings, J. (2023). How ChatGPT will change software engineering education. Proceedings of the 2023 Conference on Innovation and Technology in Computer Science Education V. 1 (pp. 110-116). :
Dellermann, D., Calma, A., Lipusch, N., Weber, T., Weigel, S., & Ebel, P. (2021). The Future of Human-AI Collaboration: a Taxonomy of Design Knowledge for Hybrid Intelligence Systems. arXiv preprint arXiv:2105.03354. https://doi.org/10.48550/arXiv.2105.03354 :
https://doi.org/10.48550/arXiv.2105.03354
Fergus, S., Botha, M., & Ostovar, M. (2023). Evaluating Academic Answers Generated Using ChatGPT. Journal of Chemical Education, 100(4), (pp. 1672-1675). https://doi.org/10.1021/acs.jchemed.3c00087 :
https://doi.org/10.1021/acs.jchemed.3c00087
Fook, C. Y., & Sidhu, G. K. (2010). Authentic Assessment and Pedagogical Strategies in Higher Education. Journal of Social Sciences, 6(2), (pp. 153-161). https://doi.org/10.3844/jssp.2010.153.161 :
https://doi.org/10.3844/jssp.2010.153.161
Friederichs, H., Friederichs, W. J., & März, M. (2023). ChatGPT in Medical School: How Successful is AI in Progress Testing?. Medical Education Online, 28(1), 2220920. :
https://doi.org/10.1080/10872981.2023.2220920
Gamage, K. A., Dehideniya, S. C., Xu, Z., & Tang, X. (2023). ChatGPT and Higher Education Assessments: More Opportunities Than Concerns?. Journal of Applied Learning and Teaching, 6(2). :
https://doi.org/10.37074/jalt.2023.6.2.32
Grévisse, C. (2023). Comparative Quality Analysis of GPT-Based Multiple Choice Question Generation. In International Conference on Applied Informatics (pp. 435-447). Cham: Springer Nature Switzerland.:
https://doi.org/10.1007/978-3-031-46813-1_29
Guskey, T. R. (2015). On Your Mark: Challenging the Conventions of Grading and Reporting. Solution Tree Press.:
Hatmanto, E. D., & Sari, M. I. (2023). Aligning Theory and Practice: Leveraging Chat GPT for Effective English Language Teaching and Learning. E3S Web of Conferences (Vol. 440,pp. 1-18). EDP Sciences. https://doi.org/10.1051/e3sconf/202344005001. :
https://doi.org/10.1051/e3sconf/202344005001.
Heritage, M. (2022). Formative Assessment: Making It Happen in the Classroom (2nd ed.). Corwin Press. https://doi.org/10.4135/9781452219493:
https://doi.org/10.4135/9781452219493
Jonassen, D. H. (1999). Designing Constructivist Learning Environments. In C. M. Reigeluth (Ed.), Instructional-design Theories and Models: A New Paradigm of Instructional Theory (2nd ed., pp. 215–239). NY: Lawrence Erlbaum Associates.:
Kamoun, F., El Ayeb, W., Jabri, I., Sifi, S., & Iqbal, F. (2023). Knowledge, Attitude, and Perception Towards ChatGPT Among University Students and Faculty: A Preliminary Exploration. The 12th European Conference on Education (ECE2024). London. UK. (pp. 247–266). :
Kolade, O., Owoseni, A., & Egbetokun, A. (2023). Assessment on Trial? ChatGPT and the New Frontiers of Learning and Assessment in Higher Education. 01-22. Retrieved from: https://www.researchgate.net/profile/Oluwaseun-Kolade/publication/369974458_Assessment_on_trial_ChatGPT_and_the_new_frontiers_of_learning_and_assessment_in_higher_education/links/64376dc720f25554da29ad9f/Assessment-on-trial-ChatGPT-and-the-new-frontiers-of-learning-and-assessment-in-higher-education.pdf (accessed on 15 December 2023).:
Meo, S. A., Al-Masri, A. A., Alotaibi, M., Meo, M. Z. S., & Meo, M. O. S. (2023). ChatGPT Knowledge Evaluation in Basic and Clinical Medical Sciences: Multiple Choice Question Examination-based Performance. Healthcare, 11(4), (p. 2046). https://doi.org/10.3390/healthcare11142046 :
https://doi.org/10.3390/healthcare11142046
Ng, D.T.K., Leung, J.K.L., Su, J. et al. (2023). Teachers’ AI Digital Competencies and Twenty-first Century Skills in the Post-pandemic World. Education Tech Research Dev, 71, (pp. 137–161). https://doi.org/10.1007/s11423-023-10203-6 :
https://doi.org/10.1007/s11423-023-10203-6
Nikolic, S., Daniel, S., Haque, R., Belkina, M., Hassan, G. M., Grundy, S., ... & Sandison, C. (2023). ChatGPT Versus Engineering Education Assessment: A Multidisciplinary and Multi-institutional Benchmarking and Analysis of this Generative Artificial Intelligence Tool to Investigate Assessment Integrity. European Journal of Engineering Education, 48(4), (pp. 559–614). https://doi.org/10.1080/03043797.2023.2213169 :
https://doi.org/10.1080/03043797.2023.2213169
Ouyang F, Zheng L., & Jiao P (2022). Artificial Intelligence in Online Higher Education: A Systematic Review of Empirical Research from 2011 to 2020. Education and Information Technologies, 27(6), (pp. 7893-7925). https://doi.org/10.1007/s10639-022-10925-9. :
https://doi.org/10.1007/s10639-022-10925-9.
Palm, T. (2019). Performance Assessment and Authentic Assessment: A Conceptual Analysis of the Literature. Practical Assessment, Research, and Evaluation, 13(1). https://doi.org/10.7275/0qpc-ws45 :
https://doi.org/10.7275/0qpc-ws45
Prompt_Journal (2023). https://github.com/AIforU/Prompt_Journal/blob/main/Prompt%20Journal.pdf :
Rasul, T., Nair, S., Kalendra, D., Robin, M., de Oliveira Santini, F., Ladeira, W. J., ... & Heathcote, L. (2023). The Role of ChatGPT in Higher Education: Benefits, Challenges, and Future Research Directions. Journal of Applied Learning and Teaching, 6(1), (pp. 1-16). https://doi.org/10.37074/jalt.2023.6.1.29 :
https://doi.org/10.37074/jalt.2023.6.1.29
Rezwana, J., & Maher, M. L. (2023). Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-creative Systems. ACM Transactions on Computer-Human Interaction, 30(5), (pp. 1-28). https://doi.org/10.1145/3519026 :
https://doi.org/10.1145/3519026
Rybak, M., & Tytenko, S. (2023). Leveraging ChatGpt for Educational Text Analysis Aiming Assessment Generation. Modern Engineering and Innovative Technologies, (29-01), (pp. 112-116). https://doi.org/10.30890/2567-5273.2023-29-01-045 :
https://doi.org/10.30890/2567-5273.2023-29-01-045
Sidnell, J., & Stivers, T. (Eds.). (2013). The Handbook of Conversation Analysis. Wiley-Blackwell. Siemens, G. (2004). Connectivism: A Learning Theory for the Digital Age. International Journal of Instructional Technology and Distance Learning, 2(1), (pp. 3-10). https://doi.org/10.3109/0142159X.2016.1173661 :
https://doi.org/10.3109/0142159X.2016.1173661
Skrabut, S. (2023). 80 Ways to Use ChatGPT in the Classroom: Using AI to Enhance Teaching and Learning. Stan Skrabut (February 10, 2023). :
Sreedharan, S. (2023). Human-Aware AI–A Foundational Framework for Human-AI Interaction. In Proceedings of the AAAI Conference on Artificial Intelligence, 37 (13). https://doi.org/10.1609/aaai.v37i13.26822 :
https://doi.org/10.1609/aaai.v37i13.26822
Tabib, F.M., & Alrabeei, M.M. (2024). Can Guided ChatGPT Use Enhance Students’ Cognitive and Metacognitive Skills?. In: Al-Marzouqi, A., Salloum, S.A., Al-Saidat, M., Aburayya, A., Gupta, B. (eds) Artificial Intelligence in Education: The Power and Dangers of ChatGPT in the Classroom (pp. 143-154). Studies in Big Data, vol 144. Springer, Cham. https://doi.org/10.1007/978-3-031-52280-2_10 :
https://doi.org/10.1007/978-3-031-52280-2_10
Villarroel, V., Bloxham, S., Bruna, D., Bruna, C., & Herrera-Seda, C. (2018). Authentic Assessment: Creating a Blueprint for Course Design. Assessment & Evaluation in Higher Education, 43(5), (pp. 840-854). https://doi.org/10.1080/02602938.2017.1412396 :
https://doi.org/10.1080/02602938.2017.1412396
Xue, J., Hu, B., Li, L., & Zhang, J. (2022). Human—machine Augmented Intelligence: Research and Applications. Frontiers of Information Technology & Electronic Engineering, 23(8), (pp. 1139-1141). :
http://dx.doi.org/10.1631/FITEE.2250000
on June 30, 2024 - 17:51 Anonymous (not verified)
Conceive
Design
Implement
Operate
