10 Jun Working on feedback literacy with comparative judgement
Kim Borsen works for the Educational Sciences programme at Utrecht University as an undergraduate and graduate lecturer. Before joining the university, she worked as a primary school teacher and educational consultant. Together with former colleague Stephanie Kruiper, she talks about the use of Comproved in the first-year course ‘Introduction to Educational Sciences’ and how they worked on students’ feedback literacy with the tool.
How did you learn about Comproved?
Kim: ‘When I started at the university last year, Stephanie was already working with Comproved on the Introduction to Educational Sciences course. This year, I am involved in that course as course coordinator and we are working with the tool again.”
“In the course, students write an introduction to a scientific article. The topic is predetermined. It’s pretty much a set assignment. The text consists of a maximum of 2,000 words. This year, we have 150 students in the course.”
What was the motivation to start working with the tool?
Stephanie: ‘There were several reasons. We were revisiting the course anyway because the first year students who came in felt thrown in at the deep end. Students always had to write an academic paper in that course, but found it quite overwhelming. We wanted to increase the role of feedback and boost students‘ feedback literacy.’
“Comproved appealed to us because of the formative part on feedback and the summative part on more reliable assessment. We also deployed it to reduce the difference between teachers in assessment. Before, we used to make sure that we divided the papers randomly among lecturers so that you didn’t have the bias of guiding your own student. But even then, one lecturer would look at a paper.”
How did you prepare students to work with the tool?
Kim: ‘Students are introduced to Comproved in practice groups. We explain the working method at the beginning. They then set to work comparing three sample papers. This serves as a basis for jointly defining the criteria for the assignment and for getting acquainted with the principle of comparative judgement in Comproved.”
“Then they are given the task of writing an outline for their paper. They give each other peer feedback on this in Comproved. This gives them insight into how to give feedback and they get explanations from us. After that, there is a feedback round for the draft version in which both students and lecturers give feedback. Finally, they submit the final version. That is only assessed by lecturers. There is no more feedback on that either.”
Stephanie: ‘We added outline writing only in the second year we used Comproved. We noticed the first time that students only understood a bit how it worked after peer feedback on the drafts and how important it is to try your best to give good quality peer feedback. Students noticed that they depend on each other. We added the outline round so that students got more from the feedback on the concepts.”
We have been working with comparative judgement for a few years now. In another course they are also using the tool now and students really like being able to work with it again.
How do students experience the method?
Kim: ‘Some still need to get used to the method, but overall they are positive. It is nice that students can see other people’s works in Comproved. They can also see what feedback others have received. That helps because students are still a bit curious about what constitutes good feedback. We have been working with comparative judgement for a few years now. In another course they are also using the tool now and students really like being able to work with it again.”
“Students do report that the ranking causes them stress, which also causes concern among colleagues. We elaborate on how the ranking comes about and how to read it, but that doesn’t seem to help enough yet. I understood that there are different options on how to display the ranking. We are still looking within the teaching team for what is the best choice for students.”
Stephanie: ‘It becomes a kind of weighing up how far to go in this. We did deliberately choose to introduce students to this method right away. At the same time, students don’t know a lot yet and we want to teach them everything at once.”
How did the teachers experience using the tool?
Kim: ‘Last year it went very smoothly. This year, there were some more technical issues that caused noise. Some students had only received feedback once, and others saw their own paper back during comparison. We have been in touch with the Comproved team about this.”
“We also noticed that more comparisons had to be made this year due to the larger number of students, while the teaching team had not increased in size. Nobody minded, but teachers with one group of students experience relatively more revision work in relation to their hours. We asked a teacher outside the course to help assess. That teacher has to know the assignment well, which is what we ran into this year. I think you have to be more aware of how much time you spend on grading and how you divide the work load.”
How were the results of the assessment?
Stephanie: ‘It’s been a while, but previous years the results seemed similar to before we used Comproved. In my opinion, we did have fewer (severe) fails. Of course, you might wonder what that is due to. Did the students actually learn more, or did the new way of grading make us apply a different standard? Maybe it’s a combination. In any case, we do know that the assessment is now reliable.”
Some students received very extensive feedback and said ‘that must be one of you, right?’ It turned out not to be. I found it remarkable that they could no longer distinguish between student feedback and teacher feedback.
Do you notice that students have become better at giving feedback?
Kim: ‘The quality still seems very variable. You see in Comproved that students put varying amounts of time into it. With some, the feedback is therefore better than with others.”
Stephanie: ‘Sometimes students don’t realise how advanced their feedback literacy already is. Students received anonymous feedback from teachers and students. In one practice group, students came to me and said they had received certain feedback they couldn’t do anything with. I looked into it with them, and on my own screen I could see who the feedback came from. That feedback came from one of the lecturers (laughs). Of course, students would never have said they couldn’t do anything with the feedback if they had known it was teacher feedback. The opposite also happened: some students received very extensive feedback and said ‘that must be one of you, right?’. It turned out it wasn’t. I found it remarkable that they could no longer distinguish between student feedback and teacher feedback.”
What do students do with the feedback they receive?
Kim: ‘Students make an action plan to indicate what they are going to do with the feedback. I think that is very powerful. I know from last year that students really did look critically at the feedback in those action plans.”
Stephanie: ‘Last year, I had explicitly asked a few students how they viewed Comproved a year after the course. One of them said she still is critical of the feedback she receives, including teacher feedback. She does not just accept the feedback but thinks about what she herself thinks of it. The student didn’t realise it herself at all, but I saw that as a very important difference from how students often blindly do what the lecturer says.”
What do you think is the biggest advantage of the tool?
Kim: ‘That you consult more with colleagues about what quality is. There is no longer a difference between colleagues in how students are assessed. You work as a teaching team.”
Would you recommend the tool to other teachers?
Kim: ‘Yes, we have done so already. I don’t think it’s desirable for there to be differences in teacher assessments within courses. So I think comparative judgement has a real benefit. Students also see more of each other’s work and can learn a lot from that.”
“We also see benefits of using Comproved when assessing internships. Especially with these assignments, you don’t want the assessment to depend on one teacher. There is still an issue for us with how to approach this, because there is not really a fixed hand-in time.”
Do you have any tips for teachers who also want to get started with the tool?
Kim: ‘I really think you need consultation with colleagues. How do you look at things? Talk to each other beforehand about which aspects of an assignment you think are important and draw up criteria together. And I think it’s nice if you can look into how the tool works together with someone.”
Have you been inspired to get started with comparative judgement too? Read more about it in our e-book or contact us with your specific questions and we will be happy to help you get started!
