Reliable assessment with multiple teachers

Zita Albon is a lecturer in the Academic Teacher Training Programme at Utrecht University. An important part of her work deals with training academic teachers for primary education. The academic teacher training programme is a partnership between the University and the University College Utrecht, in which students complete both a degree in educational sciences and a teacher training. When she started as a teacher two years ago, she came across Comproved and was immediately enthusiastic. Now Zita also uses Comproved in another course. She finds it a great advantage that you can have a reliable assessment with multiple teachers.

Since when do you work with Comproved?

“When I started working at the university, comparative judgement was first used in the first-year course Introduction to Education Sciences. For everyone, working with Comproved was new and that was an advantage. Everyone was still learning, not just me. For me, the transition was actually easier because I had not yet developed a working method. Colleagues had to shift gears more than me. I just went along with that new structure.”

Now you also use the tool in another course?

“Yes, I started working with comparative judgement in another course after the experience in Introduction to Educational Sciences. The Orthopedagogy course is for third-year students and is taught in collaboration with the university college. We are with three lecturers. In this course, we want to make sure there is good agreement with the university college and preferably also with the placement schools where the students do the assignment. I am trying to put that collaboration more on the map.”

“In this course, for the first time, we are working with lecturers from both the university and the university college. We are also assessing together now, which is an exciting process. It is new for us to work like this. We use Comproved to keep assessing reliably with this diversity of assessors. By the way, I think that diversity is a good thing.”

How did you implement the tool in the course?

“A colleague did the implementation of Comproved in Introduction to Education Sciences. I followed along last year and took over the organisation this year, but that does not mean I am an expert. At the moment, I am not yet satisfied. This year, I was especially busy organising the course. So Orthopedagogy came a bit behind.”

“Nevertheless, I was able to use a lot of what I had learned. I used a lot that I already had: schedules, sample emails, etc. I bundled all the materials and tools I collected for working with Comproved into a folder for our department. For the master’s theses, they also wanted to experiment with comparative judgement. They are now using that folder. Furthermore, someone is now following along with me again. I think it is important that the knowledge does not lie with just one person. If we keep doing it alone every time, it will never become of all of us.”

“I did underestimate bringing new colleagues into the process. This does take some effort. I sent them an article on comparative judgement and the course manual, but I noticed not everyone reads everything. We practised comparative judgement beforehand by comparing examples. That went well. Then, once colleagues start working with comparative judgement, questions and doubts arise. They ask questions like ‘how do I know if I am looking correctly?’ In that process, I didn’t really guide and there wasn’t always time for that. So I find that it is important to put Comproved on the agenda of the teachers’ meeting every week.”

“Colleagues are all involved in teaching and education science. At some point, some get really interested and start asking quite in-depth and technical questions. I read the manual and blogs on the website, but I can’t always find all the information or it remains too technical for me too. That’s a danger I think. Ideally, you want to have an expert on your team who also has the time to dig deeper. The people at Comproved are very approachable, though. So if I don’t understand something or if something goes wrong, I can easily e-mail someone. That’s nice.”

What do the students think of Comproved?

“In Orthopedagogy, we have a group of students who are encountering Comproved for the second time, because they have already used it in Introduction to Educational Sciences in the first year. We use Comproved both for feedback and for summative assessment of an advisory assignment. Giving feedback themselves they found very exciting. As teachers, we participate in giving feedback, but students cannot see which feedback comes from the teacher and which from peers. There were students who wanted to know exactly which feedback came from the lecturer. I said to them, ‘the fact that you can’t see which feedback comes from the lecturer says it all’. The distinction between teacher and peer feedback really couldn’t be seen. They gave each other very good feedback.”

“We are not yet using Comproved for all assignments in the course. We had one assignment where we felt Comproved was less suitable and so here we did not use it. The students regretted that and asked to do all assignments in Comproved. They said: ‘we can do that, we understand the programme, we give good feedback and get much more feedback. So why should we give 1-on-1 feedback now?’ They were right too.”

“Practically, we still ran into some things. For instance, in our case Comproved is not embedded in the Learning Management System. Plagiarism control is also still separate from Comproved. Students have to be careful where they submit their work, which sometimes causes problems. It also creates quite a bit of regulatory work.”

How did you experience the summative assessment?

“Before, we used a rubric for summative assessment. This was perceived as complicated by students. We are now almost done comparing for the final assessment and the ranking is reliable. I was worried whether the reliability would be high enough, because one teacher really did have a different view. But it went well. Because it is a relatively small course with only 23 students, I do notice that it is not so anonymous now. I read a piece and know which student it is. Students sometimes still put their name on the piece even though we tell them not to do so. I prefer to compare completely anonymously.”

“For determining grades, we have added a sheet that says ‘pass’. So we occasionally compared the word ‘pass’ with a paper to set the standard together. The ‘pass came’ in at spot 10 in the ranking. I thought: oh no, 14 fails is really a lot. We are still going to determine the marks by grading two papers on the ranking, after which Comproved will calculate the rest of the marks. I took a quick look at those two papers and I think the pass sheet is not in the right place. I hope my colleagues think so too. At the same time, colleagues already said that there are often a lot of fails in this course. Students also found it a difficult assignment. We need to discuss it some more.”

What do you find the biggest advantage of Comproved?

“That you make good use of that diversity of teachers and therefore assess much more reliably with each other. I’m sure we really assess differently. That says nothing about teachers and quality, we are just so different. I think it’s good that you can look at things differently and still give students a reliable assessment. I do like that.”

“On the student side, the advantage is that they see a lot more examples, get feedback and are not so insecure. In Introduction to Educational Sciences in the first year, they see three examples, so they immediately get a picture of what it should look like. Then they go into their process differently than if you don’t show anything, which still happens quite often in some courses. Teachers don’t give examples because ‘what use is that to them?’ I didn’t give examples in this course, but that was because we revised the assignment. Something new was added and we don’t have any examples of that yet. Another factor is that this course is so small and the student group knows each other well. Examples from last year are sometimes recognised. That anonymity remains difficult.”

Would you recommend the tool to other teachers?

“Yes, mainly because of that reliability. I have recommended the tool to several people. At psychology, for example. I haven’t heard yet whether they will start working with it. A colleague had previously promoted the tool to our teaching team, so at the master’s thesis they are also starting to experiment. They are tackling that well. The lecturers there find it difficult. They say ‘you can’t compare a thesis in Comproved’. The coordinator is going to try it anyway. They are going to run a pilot first, also to bring more teachers along.”

“One colleague was a bit sceptical. She thought you can’t compare assignments and give peer feedback in this way with first-year teachers because they are often too insecure. I then deliberately invited that colleague as an assessor in the course. It did help. She did not completely change her mind because she herself was not part of the course but only participated in the assessment. As a result, she did not see the whole process. It’s really not like we throw students in the deep end. They get a lot of guidance during classes. That guidance focuses on how to give feedback, how to get that writing process going… This colleague didn’t have that information. I think it’s important to bring colleagues along into this new way of working.”

Do you have tips for teachers who want to work with Comproved?

“Yes, quite a list (laughs). Especially taking your teaching team along. People are going to put their own interpretation on it anyway, and this creates misconceptions. I heard again that colleagues started writing down points during the comparison, instead of looking at it holistically. Furthermore, I saw in Comproved that someone was comparing for a very long time. That new colleague took 45 minutes on her first comparison. She had to do 80. I said ‘that’s not possible, do the math, you’re going to have to do it differently’. Then we do give each other tips on how to do it. For instance, some papers have too many headings, that says a lot already. So just looking at it in a different way.”

“I also always say: take a timer and don’t look at a comparison for longer than a certain time. However, I’am slow at comparing and I don’t want it faster than I’m doing it now. Funny enough, the people who were super fast are still reliable. Trust each other’s judgement. Everyone is looking at something they care about.”

“Working with Comproved means we have to discuss a lot. But isn’t that actually better? Because apparently we never discuss when we assess. Only the people who are in doubt about their judgement go and ask a colleague for a second opinion, but it is not standard that you look at a paper with two people. Now we all look at the same papers together. The fact that you have to tune in with each other is sometimes felt to be a burden. However, since you look at it together, the assessment is of better quality.”

Have you been inspired to get started with the comparing tool yourself? Read more in our e-book or contact us!