This paper reports a method for discerning formative assessment categories and criteria from tacit expert practice for the evaluation of reflective learning diaries with AI, and a taxonomy that covers the main dimensions, analytical frames, and criteria discovered by using the method. The approach is rooted in over two decades of practice in the context of reflective education in design and engineering. The reported category structure surfaced in one instructor’s practice across 2023-2025 through a hermeneutic procedure, where the AI-based assessment tooling was constantly improved by comparing the instructor’s and AI’s assessments of 1346 individual diary entries from in total 164 students across three project-course instances. The evolution of the instruction for the AI happened mainly through disagreements where the instructor was not satisfied with the AI’s assessment, which led into a rewrite of the instructions in the AI’s system prompt. This process made visible and shareable a formerly tacitly operating set of frames and criteria that the instructor utilised in their work. The approach shows a path to enabling a formative assessment practice of students reflective learning diaries that can result in more elaborate, relevant, and consistent assessments than those provided by human evaluators without AI.
