This article presents a mechanism for monitoring a program's effectiveness and efficiency in reaching its intended educational goals across the curriculum. Starting in 2011, the School of Engineering at the Universidad Católica de la Santísima Concepción (UCSC) has been implementing a curricular reform based on the CDIO approach. Since then, every two years we have performed a self-evaluation of the Computer Science program’s compliance with the CDIO Initiative standards. Along the way, we have developed tools and mechanisms to systematize the gathering of data for these continuous evaluation processes.
In particular, we have defined three intermediate milestones at which to measure the achievement of student learning outcomes associated with CDIO Syllabus levels 2, 3 and 4: at the end of the second semester, at the end of the sixth semester and at the end of the ninth semester. At each milestone, we are interested not only in analyzing data associated with traditional course-level grades, but also in how student achievement levels are perceived by the students and the instructors.
In order to gather the perception of student achievement levels in both students and instructors, we used sample surveys similar to the one presented in Appendix H of the CDIO Syllabus (Crawley, 2001), where the proficiency levels are:
1. To have experienced or been exposed to, 2. To be able to participate in and contribute to, 3. To be able to understand and explain, 4. To be skilled in the practice or implementation, 5. To be able to lead or innovate in
Then, we measure the gaps between these perceptions and present them in easy-to-understand radial graphs. This information is used to detect low proficiency levels and to manage instructors and students expectations of their proficiency levels at each milestone.
Also, students are required to have completed a summer internship by the end of the ninth semester. Students must complete a self-evaluation performance survey regarding their internship, while employers must evaluate the student’s performance. These surveys focus on personal and interpersonal skills and attitudes. Again, we measure the gaps between these results and present them in easy-to-understand radial graphs. This information is used in a similar manner as described before.
This mechanism has been applied every semester since 2015. Our preliminary results have shown that student and instructor perception of proficiency levels show a greater variance for the first two milestones. However, by the end of the ninth semester, these gaps have been reduced, which shows that both students and instructors have a better understanding of their proficiency levels. Moreover, students are slightly more critical of their competence levels than their instructors and employers.
Proceedings of the 13th International CDIO Conference in Calgary, Canada, June 18-22 2017