Patrick van Lommel is the pioneer in the use of the Comproved comparing tool in Avans University of Applied Sciences. He has also introduced the tool to his fellow lecturers so that it has now become an established method. In this interview, he shares his experiences with the tool and comparative judgement.

Why did you decide to start using the comparing tool?

“I decided to use the tool because, on the one hand, I became frustrated with trying to describe test criteria and rubrics as accurately as possible. We have been working on these in vain for years, but we have found that they are never applicable to all papers.
Since the very beginning, I have been able to encourage my colleagues to use the tool as well. I managed to convince them quite quickly because they were struggling with the same frustrations as me. These could be remedied using comparative judgement.”

“Since we started assessing with the comparing tool, we have been very consistent in our judgments. This gives us peace of mind and at the same time increases the reliability of the assessments. we are therefore very satisfied with the way we can now formulate the assignments with the corresponding criteria lists. This is because we now first seek a consensus together among the criteria of all the assessors. Then these criteria lists are reformulated and summarized. Because everyone now agrees on the criteria that students must meet to achieve a good result, I feel they are clear, concise and easy to understand for both students and teachers.”

What is the added value of Comproved?

“I find the fact that Comproved takes a different approach via comparative judgement advantageous for several reasons. First of all, the reliability is much higher since using the tool. For one, this is because the works are now assessed anonymously. This prevents students from looking for their friend’s work to then give them a favorable grade. On the other hand, reliability also increases because there are multiple assessors comparing the same works multiple times. This provides a very reliable underlying basis of the final grade which is advantageous for both the student and the teacher. In addition, the method is very easy and natural because you only have to say which of two pieces of work is the best and thus there is no immediate need to give a score.”

“Second: Using the comparing tool created a kind of self-reflection that allowed us to form an understanding of our own ways of assessing. This revealed that the preferences between assessors were quite far apart. A finding that we might otherwise never have discovered since everyone assesses for themselves and thus there is no interaction between different assessors.
We explored this in more detail afterwards. On the one hand to establish the final consensus, on the other hand to list all the feedback that was given and to analyze exactly what criteria each evaluator was paying attention to. Based on this information, the assignment was reformulated so that it is now sharper and clearer. As a result, the students have a better idea of how to achieve a better result.”

Do you also use the tool for other applications?

“Definitely. We also use the tool as a formative assessment tool in the first and final year. In the first year, we present the students with some works from the previous year. The students then have to use the tool to indicate the best of two products each time and provide feedback on why they think these are the best.
After collecting all the feedback, students can form their own test criteria and think critically about the quality of the final product. When these test criteria are subsequently compared with the predetermined criteria, it turns out that they are not that far apart. In short, the students thus discover by themselves what criteria are important and what ultimately leads to a quality result.”

“In the final year, on the other hand, the students can conduct their own peer assessment with their preliminary thesis works using the tool. In this way, students can compare each other’s work which also leads to a better understanding regarding the quality of the final product.”

Want to get started with comparative judgement yourself? Find more info about the tool here.